You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by silvade <st...@gmail.com> on 2015/10/26 13:40:37 UTC
Message getting stuck on topic, leading to KahaDB log files not
being deleted and disk running out of space.
Hi,
I am running into a problem where messages are getting stuck my topic,
leading to KahaDB log files not being deleted and disk running out of space.
The setup seem to work fine for a week or so. When my durable subscribers go
offline their messages are removed from the data.*.log after the configured
5 minutes. After a week or so I start getting errors similar to the one
below:
2015-10-26 09:12:59,090 | TRACE | not removing data file: 573 as contained
ack(s) refer to referenced file: [572, 573] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
Eventually activemq runs out of disk space since these journal files are not
removed. In order to get ActiveMq started again I remove all the files in
kahadb (e.g. db-*.log, db.data, etc ...) kahadb data files and restart
activeMq.
My setup is a follows:
(1) ActiveMq.xml (v5.12.0) with kahadb:
<persistenceAdapter>
<kahaDB directory="${activemq.base}/data/kahadb"
ignoreMissingJournalfiles="true"
checkForCorruptJournalFiles="true"
checksumJournalFiles="true" />
</persistenceAdapter>
...
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic="data" expireMessagesPeriod="30000">
<deadLetterStrategy>
<sharedDeadLetterStrategy processExpired="false" />
</deadLetterStrategy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
(2) Durable subscribers configuration in spring web application
...
<bean id="amqConnectionFactory"
class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="transportListener"
ref="myActiveMQExceptionListener"/>
<property name="brokerURL"
value="failover:(tcp://localhost:61616)?maxReconnectDelay=1000&startupMaxReconnectAttempts=1&jms.prefetchPolicy.all=100&jms.useAsyncSend=true"
/>
<property name="clientID" value="myClient" />
</bean>
...
(3) Camel.xml - Camel routes to connect and consume message from a remote
topic
...
<camelContext xmlns="http://camel.apache.org/schema/spring" id="camel">
<endpoint id="endpoint" uri="dev:topic:local?timeToLive=300000">
</endpoint>
<route>
<from uri="remote:topic:remote_data"/>
<to ref="endpoint"/>
</route>
</camelContext>
<bean id="remote"
class="org.apache.activemq.camel.component.ActiveMQComponent">
<property name="brokerURL" value="failover:tcp://XX.XX.XX.XX:61616"/>
</bean>
Has anyone experienced a similar problem? Any help would really be
appreciated.
Thanks in advance,
Steve
--
View this message in context: http://activemq.2283324.n4.nabble.com/Message-getting-stuck-on-topic-leading-to-KahaDB-log-files-not-being-deleted-and-disk-running-out-of-tp4703370.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Message getting stuck on topic, leading to KahaDB log files not
being deleted and disk running out of space.
Posted by Christopher Shannon <ch...@gmail.com>.
I believe that the expiration task should go ahead and mark the message as
acknowledged if it is expired for each subscription which should mean that
message would no longer prevent a file from being garbage collected. One
thing you can do is you can turn on debug logging for KahaDB to try and
determine why it's not garbage collecting your log files. If you turn on
trace logging for org.apache.activemq.store.kahadb.MessageDatabase, it will
show some information about the files it is checking during the garbage
collection task.
On Tue, Oct 27, 2015 at 6:36 AM, silvade <st...@gmail.com> wrote:
> Hi Christopher, thanks for quick responds.
>
> I have some further information regarding my setup.
>
> Regarding the client ID and subscription ID, I am using the same value
> (i.e.
> "client-1") as the clientID and durableSubscriptionName.
>
> My applicationContext.xml file has set this as follows:
>
>
> <bean id="amqConnectionFactory"
> class="org.apache.activemq.ActiveMQConnectionFactory">
> <property name="clientID" value="client-1" />
> </bean>
> ...
> <bean id="darwinConsumer"
> class="org.springframework.jms.listener.DefaultMessageListenerContainer"
> destroy-method="stop">
> <property name="autoStartup" value="false"/>
> <property name="concurrentConsumers" value="1"/>
> <property name="maxConcurrentConsumers" value="1"/>
> <property name="connectionFactory" ref="jmsConnectionFactory" />
> <property name="exceptionListener" ref="jmsConnectionFactory"/>
> <property name="sessionTransacted" value="true" />
> <property name="subscriptionDurable" value="true"/>
> <property name="durableSubscriptionName" value="client-1" />
> <property name="destinationName" value="local" />
> <property name="messageListener" ref="messageListenerAdapter" />
> <property name="recoveryInterval" value="1000"
> <property name="sessionAcknowledgeModeName"
> value="AUTO_ACKNOWLEDGE" />
> <property name="clientId" value="client-1" />
> </bean>
> ...
>
>
> I have noticed that the journal data in the kahadb directory is not removed
> even after 48 hours. I was under the impression that when a message has
> expired in the kahadb directory (e.g. db-X.log) , a job would run which
> would remove the file (even if the durable subscriber was offline at the
> time). It appears that the db-X.log files are not being removed even though
> all the messages in them have expired and are eligible for removal.
>
> Thanks,
> Steve
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Message-getting-stuck-on-topic-leading-to-KahaDB-log-files-not-being-deleted-and-disk-running-out-of-tp4703370p4703402.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Message getting stuck on topic, leading to KahaDB log files not
being deleted and disk running out of space.
Posted by silvade <st...@gmail.com>.
Hi Christopher, thanks for quick responds.
I have some further information regarding my setup.
Regarding the client ID and subscription ID, I am using the same value (i.e.
"client-1") as the clientID and durableSubscriptionName.
My applicationContext.xml file has set this as follows:
<bean id="amqConnectionFactory"
class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="clientID" value="client-1" />
</bean>
...
<bean id="darwinConsumer"
class="org.springframework.jms.listener.DefaultMessageListenerContainer"
destroy-method="stop">
<property name="autoStartup" value="false"/>
<property name="concurrentConsumers" value="1"/>
<property name="maxConcurrentConsumers" value="1"/>
<property name="connectionFactory" ref="jmsConnectionFactory" />
<property name="exceptionListener" ref="jmsConnectionFactory"/>
<property name="sessionTransacted" value="true" />
<property name="subscriptionDurable" value="true"/>
<property name="durableSubscriptionName" value="client-1" />
<property name="destinationName" value="local" />
<property name="messageListener" ref="messageListenerAdapter" />
<property name="recoveryInterval" value="1000"
<property name="sessionAcknowledgeModeName"
value="AUTO_ACKNOWLEDGE" />
<property name="clientId" value="client-1" />
</bean>
...
I have noticed that the journal data in the kahadb directory is not removed
even after 48 hours. I was under the impression that when a message has
expired in the kahadb directory (e.g. db-X.log) , a job would run which
would remove the file (even if the durable subscriber was offline at the
time). It appears that the db-X.log files are not being removed even though
all the messages in them have expired and are eligible for removal.
Thanks,
Steve
--
View this message in context: http://activemq.2283324.n4.nabble.com/Message-getting-stuck-on-topic-leading-to-KahaDB-log-files-not-being-deleted-and-disk-running-out-of-tp4703370p4703402.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Message getting stuck on topic, leading to KahaDB log files not
being deleted and disk running out of space.
Posted by Christopher Shannon <ch...@gmail.com>.
Nevermind, I just double checked the expiration task. It will only browse
up to 400 messages, but expired messages are not counted in that number so
that probably isn't the issue.
On Mon, Oct 26, 2015 at 12:35 PM, Christopher Shannon <
christopher.l.shannon@gmail.com> wrote:
> How fast are messages being sent to the topic? By default, the expire
> messages task only pages in the max browse page size (which defaults to 400
> messages) before stopping. So one thing to check would be if your incoming
> message rate is faster than the broker can keep up with expiration.
>
> On Mon, Oct 26, 2015 at 8:40 AM, silvade <st...@gmail.com> wrote:
>
>> Hi,
>>
>> I am running into a problem where messages are getting stuck my topic,
>> leading to KahaDB log files not being deleted and disk running out of
>> space.
>>
>> The setup seem to work fine for a week or so. When my durable subscribers
>> go
>> offline their messages are removed from the data.*.log after the
>> configured
>> 5 minutes. After a week or so I start getting errors similar to the one
>> below:
>> 2015-10-26 09:12:59,090 | TRACE | not removing data file: 573 as contained
>> ack(s) refer to referenced file: [572, 573] |
>> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> Checkpoint Worker
>>
>> Eventually activemq runs out of disk space since these journal files are
>> not
>> removed. In order to get ActiveMq started again I remove all the files in
>> kahadb (e.g. db-*.log, db.data, etc ...) kahadb data files and restart
>> activeMq.
>>
>>
>> My setup is a follows:
>> (1) ActiveMq.xml (v5.12.0) with kahadb:
>>
>> <persistenceAdapter>
>> <kahaDB directory="${activemq.base}/data/kahadb"
>> ignoreMissingJournalfiles="true"
>> checkForCorruptJournalFiles="true"
>> checksumJournalFiles="true" />
>> </persistenceAdapter>
>>
>> ...
>> <destinationPolicy>
>> <policyMap>
>> <policyEntries>
>> <policyEntry topic="data" expireMessagesPeriod="30000">
>> <deadLetterStrategy>
>> <sharedDeadLetterStrategy
>> processExpired="false" />
>> </deadLetterStrategy>
>> </policyEntry>
>> </policyEntries>
>> </policyMap>
>> </destinationPolicy>
>>
>> (2) Durable subscribers configuration in spring web application
>> ...
>> <bean id="amqConnectionFactory"
>> class="org.apache.activemq.ActiveMQConnectionFactory">
>> <property name="transportListener"
>> ref="myActiveMQExceptionListener"/>
>> <property name="brokerURL"
>>
>> value="failover:(tcp://localhost:61616)?maxReconnectDelay=1000&startupMaxReconnectAttempts=1&jms.prefetchPolicy.all=100&jms.useAsyncSend=true"
>> />
>> <property name="clientID" value="myClient" />
>> </bean>
>> ...
>>
>> (3) Camel.xml - Camel routes to connect and consume message from a remote
>> topic
>>
>> ...
>> <camelContext xmlns="http://camel.apache.org/schema/spring" id="camel">
>>
>> <endpoint id="endpoint" uri="dev:topic:local?timeToLive=300000">
>> </endpoint>
>>
>> <route>
>> <from uri="remote:topic:remote_data"/>
>> <to ref="endpoint"/>
>> </route>
>>
>> </camelContext>
>>
>>
>> <bean id="remote"
>> class="org.apache.activemq.camel.component.ActiveMQComponent">
>> <property name="brokerURL" value="failover:tcp://XX.XX.XX.XX:61616"/>
>> </bean>
>>
>>
>> Has anyone experienced a similar problem? Any help would really be
>> appreciated.
>>
>> Thanks in advance,
>> Steve
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/Message-getting-stuck-on-topic-leading-to-KahaDB-log-files-not-being-deleted-and-disk-running-out-of-tp4703370.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>
Re: Message getting stuck on topic, leading to KahaDB log files not
being deleted and disk running out of space.
Posted by Christopher Shannon <ch...@gmail.com>.
How fast are messages being sent to the topic? By default, the expire
messages task only pages in the max browse page size (which defaults to 400
messages) before stopping. So one thing to check would be if your incoming
message rate is faster than the broker can keep up with expiration.
On Mon, Oct 26, 2015 at 8:40 AM, silvade <st...@gmail.com> wrote:
> Hi,
>
> I am running into a problem where messages are getting stuck my topic,
> leading to KahaDB log files not being deleted and disk running out of
> space.
>
> The setup seem to work fine for a week or so. When my durable subscribers
> go
> offline their messages are removed from the data.*.log after the configured
> 5 minutes. After a week or so I start getting errors similar to the one
> below:
> 2015-10-26 09:12:59,090 | TRACE | not removing data file: 573 as contained
> ack(s) refer to referenced file: [572, 573] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
>
> Eventually activemq runs out of disk space since these journal files are
> not
> removed. In order to get ActiveMq started again I remove all the files in
> kahadb (e.g. db-*.log, db.data, etc ...) kahadb data files and restart
> activeMq.
>
>
> My setup is a follows:
> (1) ActiveMq.xml (v5.12.0) with kahadb:
>
> <persistenceAdapter>
> <kahaDB directory="${activemq.base}/data/kahadb"
> ignoreMissingJournalfiles="true"
> checkForCorruptJournalFiles="true"
> checksumJournalFiles="true" />
> </persistenceAdapter>
>
> ...
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry topic="data" expireMessagesPeriod="30000">
> <deadLetterStrategy>
> <sharedDeadLetterStrategy
> processExpired="false" />
> </deadLetterStrategy>
> </policyEntry>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
>
> (2) Durable subscribers configuration in spring web application
> ...
> <bean id="amqConnectionFactory"
> class="org.apache.activemq.ActiveMQConnectionFactory">
> <property name="transportListener"
> ref="myActiveMQExceptionListener"/>
> <property name="brokerURL"
>
> value="failover:(tcp://localhost:61616)?maxReconnectDelay=1000&startupMaxReconnectAttempts=1&jms.prefetchPolicy.all=100&jms.useAsyncSend=true"
> />
> <property name="clientID" value="myClient" />
> </bean>
> ...
>
> (3) Camel.xml - Camel routes to connect and consume message from a remote
> topic
>
> ...
> <camelContext xmlns="http://camel.apache.org/schema/spring" id="camel">
>
> <endpoint id="endpoint" uri="dev:topic:local?timeToLive=300000">
> </endpoint>
>
> <route>
> <from uri="remote:topic:remote_data"/>
> <to ref="endpoint"/>
> </route>
>
> </camelContext>
>
>
> <bean id="remote"
> class="org.apache.activemq.camel.component.ActiveMQComponent">
> <property name="brokerURL" value="failover:tcp://XX.XX.XX.XX:61616"/>
> </bean>
>
>
> Has anyone experienced a similar problem? Any help would really be
> appreciated.
>
> Thanks in advance,
> Steve
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Message-getting-stuck-on-topic-leading-to-KahaDB-log-files-not-being-deleted-and-disk-running-out-of-tp4703370.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>