You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by petrk <pk...@yahoo.com> on 2014/09/26 17:57:54 UTC
Managing persistent store space
Hi
I have a single instance of ActiveMQ v5.9 using persistent store and we are
constantly running out of space.
We use persistent messages with KahaDB and have 50gb store defined.
The queues directory only actually contains 26GB, but ActiveMQ admin says
"Store percent used 94", is that right?
Browsing the number of pending messages in queues, there is no way there is
26GB or 50GB worth of data!
I have noticed that we have lots of ActiveMQ.Advisory.Consumer.Queue.*
topics, which we don't use, but we use the advisoryWhenFull="true" in
conjuction with sendFailIfNoSpace="true" so that we get notified when
messages can't be queued. We also expire messages after 30 days, in hope
they will get deleted. If I disable advisory message am I going to lose
these two functions?
Any ideas?
Here are the relevant bits from our activemq.xml:
<bean id="store"
class="org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter" >
<property name="directory" value="/opt/xyz/queues" />
<property name="journalMaxFileLength" value="3145728" />
<property name="checkpointInterval" value="5000" />
<property name="cleanupInterval" value="5000" />
</bean>
<broker xmlns="http://activemq.apache.org/schema/core"
useJmx="true" brokerName="localhost" dataDirectory="/opt/xyz/queues"
persistenceAdapter="#store">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">">
<deadLetterStrategy>
<sharedDeadLetterStrategy processExpired="false"/>
</deadLetterStrategy>
</policyEntry>
<policyEntry queue=">" cursorMemoryHighWaterMark="90" maxPageSize="10000"
queuePrefetch="10000" producerFlowControl="false" advisoryWhenFull="true">
<pendingQueuePolicy>
<storeCursor/>
</pendingQueuePolicy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<systemUsage>
<systemUsage sendFailIfNoSpace="true">
<memoryUsage>
<memoryUsage limit = "8 gb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="50 gb" store="#store"/>
</storeUsage>
<tempUsage>
<tempUsage limit="10 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<plugins>
<timeStampingBrokerPlugin
zeroExpirationOverride="2419200000"/>
<discardingDLQBrokerPlugin dropAll="true"
dropTemporaryTopics="true" dropTemporaryQueues="true" />
</plugins>
</broker>
--
View this message in context: http://activemq.2283324.n4.nabble.com/Managing-persistent-store-space-tp4685913.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Managing persistent store space
Posted by petrk <pk...@yahoo.com>.
Right I have managed to set up Multi kahaDB persistence adapter from inside
of the broker config:
<bean id="mainStore"
class="org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter" >
<property name="journalMaxFileLength" value="3145728" />
<property name="checkpointInterval" value="5000" />
<property name="cleanupInterval" value="5000" />
<property name="directory" value="/opt/xyz/queues" />
</bean>
<bean id="otherStore"
class="org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter" >
<property name="journalMaxFileLength" value="3145728" />
<property name="checkpointInterval" value="5000" />
<property name="cleanupInterval" value="5000" />
<property name="directory" value="/opt/xyz/queues/Other" />
</bean>
<bean id="filteredMainStore"
class="org.apache.activemq.store.kahadb.FilteredKahaDBPersistenceAdapter" >
<property name="persistenceAdapter" ref="mainStore" />
</bean>
<bean id="filteredOtherStore"
class="org.apache.activemq.store.kahadb.FilteredKahaDBPersistenceAdapter" >
<property name="persistenceAdapter" ref="otherStore" />
<property name="queue" value="Other*" />
</bean>
<bean id="store"
class="org.apache.activemq.store.kahadb.MultiKahaDBPersistenceAdapter" >
<property name="directory" value="/opt/xyz/queues" />
<property name="filteredPersistenceAdapters">
<list>
<ref bean="filteredOtherStore" />
<ref bean="filteredMainStore" />
</list>
</property>
</bean>
However, the existing queues before the above persistance adapter changes
are no longer visible/don't exist. I need to be able to keep the existing
queues if changing to this MultiKahaDBPersistenceAdapter, is there any way
to do that?
when activeMQ starts up I can see the following log entry:
2014-10-01 16:16:35,840 | INFO | Using Persistence Adapter:
MultiKahaDBPersistenceAdapter[C:\apache-activemq-5.9.0\bin\activemq-data\mKahaDB][KahaDBPersistenceAdapter[C:\opt\RaxStore\queues\StoreLogon\queue#3a#2f#2fStoreLogon#2a],
KahaDBPersistenceAdapter[C:\opt\RaxStore\queues\0]] |
org.apache.activemq.broker.BrokerService | main
--
View this message in context: http://activemq.2283324.n4.nabble.com/Managing-persistent-store-space-tp4685913p4686058.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Managing persistent store space
Posted by petrk <pk...@yahoo.com>.
I have one issue though, as per Limiting disk store usage with Active MQ
<http://open-source-adventures.blogspot.co.uk/2009/01/limiting-disk-store-usage-with-active.html>
I need to explicitly specify a persistence adapter as a bean within the
configuration file and reference it from the broker and the storage
definition otherwise the store size gets ignored.
and apperars I can only specify Multi kahaDB persistence adapter from inside
of the broker config.
Has anyone tried to combine Multi kahaDB persistence adapter whilst
specifying store usage?
--
View this message in context: http://activemq.2283324.n4.nabble.com/Managing-persistent-store-space-tp4685913p4686046.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Managing persistent store space
Posted by Andreas Gies <an...@wayofquality.de>.
Hi again,
as far as I understood (and believe I have observed it in our case) you
can use wildcards
to qualify the kahadb instance. The instance with no destination
qualification is the default
kahadb instance.
Best regards
Andreas
On 29/09/14 14:56, petrk wrote:
> That is an interesting solution as we do have two streams of messages on our
> queues, fast one with lots of data and a slow one with more smaller sized
> messages.
>
> Does anyone know if the kaha db adapter supports use of wildcards (i.e.
> "OrdersForDevice*"), as we have dynamic queuenames. would something like
> this work:
>
> <persistenceAdapter>
> <mKahaDB directory="${activemq.base}/data/kahadb">
> <filteredPersistenceAdapters>
> <filteredKahaDB queue="OrdersForDevice*">
> <persistenceAdapter>
> <kahaDB />
> </persistenceAdapter>
> </filteredKahaDB>
> <filteredKahaDB>
> <persistenceAdapter>
> <kahaDB />
> </persistenceAdapter>
> </filteredKahaDB>
> </filteredPersistenceAdapters>
> </mKahaDB>
> </persistenceAdapter>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Managing-persistent-store-space-tp4685913p4685981.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
Andreas Gies
WoQ -- Way of Quality GmbH
Geschäftsführer & CTO
/eMail:/andreas@wayofquality.de <ma...@wayofquality.de>
/Tel:/ +49 151 23470823
/Fax:/ +49 1805 006534 2114
/Twitter:/ andreasgies /Skype:/ giessonic
/LinkedIn:/ <http://de.linkedin.com/pub/andreas-gies/0/594/aa5/>
(http://de.linkedin.com/pub/andreas-gies/0/594/aa5/)
/Xing:/ <http://www.xing.com/profile/Andreas_Gies>
(http://www.xing.com/profile/Andreas_Gies)
/Blog:/ <http://www.wayofquality.de/index.php/en/blog>
(http://www.wayofquality.de/index.php/en/blog)
/Github:/ <https://github.com/atooni> (https://github.com/atooni)
/Amtsgericht Landshut:/HRB 8352//
//
/Ust.-Id.:/ DE274771254
Haftungsausschluss
Diese Email kann vertrauliche und/oder rechtlich geschützte
Informationen enthalten und ist ausschließlich für den/die benannten
Adressaten bestimmt. Sollten Sie nicht der beabsichtigte Empfänger sein
oder diese Email irrtümlich erhalten haben, ist es Ihnen nicht gestattet
diese Mail oder einen Teil davon ohne unsere Erlaubnis zu verbreiten, zu
kopieren, unbefugt weiterzuleiten oder zu behalten. Informieren Sie
bitte sofort den Absender telefonisch oder per Email und löschen Sie
diese Email und alle Kopien aus Ihrem System. Wir haften nicht für die
Unversehrtheit von Emails, nachdem sie unseren Einflussbereich verlassen
haben.
Disclaimer
This email may contain confidential and/or privileged information and is
intended solely for the attention and use of the named addressee(s). If
you are not the intended recipient, or a person responsible for
delivering it to the intended recipient, you are not authorized to and
must not disclose, copy, distribute, or retain this message or any part
of it without our authority. Please contact the sender by call or reply
email immediately and destroy all copies and the original message. We
are not responsible for the integrity of emails after they have left our
sphere of control.
//
Re: Managing persistent store space
Posted by petrk <pk...@yahoo.com>.
That is an interesting solution as we do have two streams of messages on our
queues, fast one with lots of data and a slow one with more smaller sized
messages.
Does anyone know if the kaha db adapter supports use of wildcards (i.e.
"OrdersForDevice*"), as we have dynamic queuenames. would something like
this work:
<persistenceAdapter>
<mKahaDB directory="${activemq.base}/data/kahadb">
<filteredPersistenceAdapters>
<filteredKahaDB queue="OrdersForDevice*">
<persistenceAdapter>
<kahaDB />
</persistenceAdapter>
</filteredKahaDB>
<filteredKahaDB>
<persistenceAdapter>
<kahaDB />
</persistenceAdapter>
</filteredKahaDB>
</filteredPersistenceAdapters>
</mKahaDB>
</persistenceAdapter>
--
View this message in context: http://activemq.2283324.n4.nabble.com/Managing-persistent-store-space-tp4685913p4685981.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Managing persistent store space
Posted by Andreas Gies <an...@wayofquality.de>.
Hi
we had a similar issue in our ActiveMQ broker. The reason was that we
had a topic that we use for guaranteed message delivery with message
persistence and durable subscribers. Some durable subscribers might be
offline and we spotted that even with a few MB worth of messages our
persistent store was growing to an unexplainable size.
The reason underneath was that we were using one kahadb instance for ALL
our messaging traffic. Since kahadb has a write only journal, the worst
case would be that our messages waiting for delivery would be spread
across journal files interleaved with all the other journal entries that
refer to already processed messages.
Gary has written a very good blog entry that describes our problem
nicely :
http://blog.garytully.com/2011/11/activemq-multiple-kahadb-instances.html
At the end we are now running with a kahadb split in 3 parts. 2 parts
serving a destination with occasinally offline consumers each and a
third part handling "the normal" JMS use case.
That solved our store problem pretty well.
For reference this is the interesting part of our config:
<amq:persistenceAdapter>
<amq:mKahaDB
directory="amq/mkahadb"
journalMaxFileLength="16m">
<amq:filteredPersistenceAdapters>
<amq:filteredKahaDB
topic="our.occasianally.disconnected.topic">
<amq:persistenceAdapter>
<amq:kahaDB
directory="amq/kahadb/topic"
journalMaxFileLength="16m"
/>
</amq:persistenceAdapter>
</amq:filteredKahaDB>
<amq:filteredKahaDB
queue="our.occasionally.disconnected.queue">
<amq:persistenceAdapter>
<amq:kahaDB
directory="amq/kahadb/queue"
journalMaxFileLength="16m"
/>
</amq:persistenceAdapter>
</amq:filteredKahaDB>
<amq:filteredKahaDB>
<amq:persistenceAdapter>
<amq:kahaDB
directory="amq/kahadb/default"
journalMaxFileLength="16m"
/>
</amq:persistenceAdapter>
</amq:filteredKahaDB>
</amq:filteredPersistenceAdapters>
</amq:mKahaDB>
</amq:persistenceAdapter>
Best regards
Andreas
On 26/09/14 17:57, petrk wrote:
> Hi
>
> I have a single instance of ActiveMQ v5.9 using persistent store and we are
> constantly running out of space.
> We use persistent messages with KahaDB and have 50gb store defined.
> The queues directory only actually contains 26GB, but ActiveMQ admin says
> "Store percent used 94", is that right?
> Browsing the number of pending messages in queues, there is no way there is
> 26GB or 50GB worth of data!
>
> I have noticed that we have lots of ActiveMQ.Advisory.Consumer.Queue.*
> topics, which we don't use, but we use the advisoryWhenFull="true" in
> conjuction with sendFailIfNoSpace="true" so that we get notified when
> messages can't be queued. We also expire messages after 30 days, in hope
> they will get deleted. If I disable advisory message am I going to lose
> these two functions?
>
> Any ideas?
>
> Here are the relevant bits from our activemq.xml:
> <bean id="store"
> class="org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter" >
> <property name="directory" value="/opt/xyz/queues" />
> <property name="journalMaxFileLength" value="3145728" />
> <property name="checkpointInterval" value="5000" />
> <property name="cleanupInterval" value="5000" />
> </bean>
>
> <broker xmlns="http://activemq.apache.org/schema/core"
> useJmx="true" brokerName="localhost" dataDirectory="/opt/xyz/queues"
> persistenceAdapter="#store">
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry topic=">">
> <deadLetterStrategy>
> <sharedDeadLetterStrategy processExpired="false"/>
> </deadLetterStrategy>
> </policyEntry>
> <policyEntry queue=">" cursorMemoryHighWaterMark="90" maxPageSize="10000"
> queuePrefetch="10000" producerFlowControl="false" advisoryWhenFull="true">
> <pendingQueuePolicy>
> <storeCursor/>
> </pendingQueuePolicy>
> </policyEntry>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
> <systemUsage>
> <systemUsage sendFailIfNoSpace="true">
> <memoryUsage>
> <memoryUsage limit = "8 gb"/>
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="50 gb" store="#store"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="10 gb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
> <plugins>
> <timeStampingBrokerPlugin
> zeroExpirationOverride="2419200000"/>
> <discardingDLQBrokerPlugin dropAll="true"
> dropTemporaryTopics="true" dropTemporaryQueues="true" />
> </plugins>
>
> </broker>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Managing-persistent-store-space-tp4685913.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
Andreas Gies
WoQ -- Way of Quality GmbH
Geschäftsführer & CTO
/eMail:/andreas@wayofquality.de <ma...@wayofquality.de>
/Tel:/ +49 151 23470823
/Fax:/ +49 1805 006534 2114
/Twitter:/ andreasgies /Skype:/ giessonic
/LinkedIn:/ <http://de.linkedin.com/pub/andreas-gies/0/594/aa5/>
(http://de.linkedin.com/pub/andreas-gies/0/594/aa5/)
/Xing:/ <http://www.xing.com/profile/Andreas_Gies>
(http://www.xing.com/profile/Andreas_Gies)
/Blog:/ <http://www.wayofquality.de/index.php/en/blog>
(http://www.wayofquality.de/index.php/en/blog)
/Github:/ <https://github.com/atooni> (https://github.com/atooni)
/Amtsgericht Landshut:/HRB 8352//
//
/Ust.-Id.:/ DE274771254
Haftungsausschluss
Diese Email kann vertrauliche und/oder rechtlich geschützte
Informationen enthalten und ist ausschließlich für den/die benannten
Adressaten bestimmt. Sollten Sie nicht der beabsichtigte Empfänger sein
oder diese Email irrtümlich erhalten haben, ist es Ihnen nicht gestattet
diese Mail oder einen Teil davon ohne unsere Erlaubnis zu verbreiten, zu
kopieren, unbefugt weiterzuleiten oder zu behalten. Informieren Sie
bitte sofort den Absender telefonisch oder per Email und löschen Sie
diese Email und alle Kopien aus Ihrem System. Wir haften nicht für die
Unversehrtheit von Emails, nachdem sie unseren Einflussbereich verlassen
haben.
Disclaimer
This email may contain confidential and/or privileged information and is
intended solely for the attention and use of the named addressee(s). If
you are not the intended recipient, or a person responsible for
delivering it to the intended recipient, you are not authorized to and
must not disclose, copy, distribute, or retain this message or any part
of it without our authority. Please contact the sender by call or reply
email immediately and destroy all copies and the original message. We
are not responsible for the integrity of emails after they have left our
sphere of control.
//