You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by kentdorsey <ke...@gmail.com> on 2006/09/09 03:33:09 UTC
Persistent messages not being cleaned up for non-durable Topic
A project that I am working on is seeing a massive build-up (over 500,000)
persistent messages for a non-durable Topic.
The configuration:
ActiveMQ 3.2.1
There is a network of brokers: A and B connected over a reliable protocol
network channel.
The brokers are being accessed as a clutser using a client connection URI
with the reliable protocol to support failover.
The topics in question have non-durable subscribers.
The messages are being published with defaults, i.e.
Message.DEFAULT_DELIVERY_MODE (persistent), Message.DEFAULT_PRIORITY,
Message.DEFAULT_TIME_TO_LIVE (no expiration).
The session acknowledgment mode is Session.AUTO_ACKNOWLEDGE.
The client behavior does not call acknowledge() on the message, since
Session.AUTO_ACKNOWLEDGE was specified.
The problems:
A slowdown in database performance :-)
Over 500,000 messages have built up in the ACTIVEMQ_MSGS table over several
months.
There are no entries in ACTIVEMS_ACKS.
The following select statement is resulting in hundreds of thousands of
"physical reads" (per the DBA) each hour:
SELECT ID, CONTAINER, MSG_ID, SENT_TO_DEADLETTER FROM ACTIVEMQ_MSGS
WHERE (EXPIRATION <> 0 AND EXPIRATION<:1)
There is no index on the EXPIRATION column. There is a JIRA issue where
this has been corrected. The index will be added to the system in question.
The questions:
What is triggering the above select statement?
I have read several reports on the forums of persistent messages not being
deleted for non-durable topics. Is there possibly a defect related to the
above configuration?
Thanks,
Kent Dorsey
--
View this message in context: http://www.nabble.com/Persistent-messages-not-being-cleaned-up-for-non-durable-Topic-tf2242336.html#a6219438
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Persistent messages not being cleaned up for non-durable Topic
Posted by kentdorsey <ke...@gmail.com>.
kentdorsey wrote:
>
>
> The problems:
>
> ...
>
> There are no entries in ACTIVEMS_ACKS.
>
This is actually not a problem, but to be expected with non-durable
subscribers :-)
--
View this message in context: http://www.nabble.com/Persistent-messages-not-being-cleaned-up-for-non-durable-Topic-tf2242336.html#a6219447
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Persistent messages not being cleaned up for non-durable Topic
Posted by kentdorsey <ke...@gmail.com>.
kentdorsey wrote:
>
> The configuration:
>
> ActiveMQ 3.2.1
> There is a network of brokers: A and B connected over a reliable protocol
> network channel.
> The brokers are being accessed as a clutser using a client connection URI
> with the reliable protocol to support failover.
> The topics in question have non-durable subscribers.
> The messages are being published with defaults, i.e.
> Message.DEFAULT_DELIVERY_MODE (persistent), Message.DEFAULT_PRIORITY,
> Message.DEFAULT_TIME_TO_LIVE (no expiration).
> The session acknowledgment mode is Session.AUTO_ACKNOWLEDGE.
> The client behavior does not call acknowledge() on the message, since
> Session.AUTO_ACKNOWLEDGE was specified.
>
Additional configuration:
The persistence is configured as cached, journaled, jdbc against an oracle
datasource.
The session is not transacted.
One topic is used to send to subscribers eith getNoLocal() == true. Another
topic is used to send to subscribers with getNoLocal() == false.
Additional information:
Both topics experience the same problem with messages never being deleted.
Proposed Workarounds:
Turn off persistence.
Set an expiration on the messages, so that the expiration column gets
calculated when they are persisted and cleanup() properly reaps them.
--
View this message in context: http://www.nabble.com/Persistent-messages-not-being-cleaned-up-for-non-durable-Topic-tf2242336.html#a6220091
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Persistent messages not being cleaned up for non-durable Topic
Posted by kentdorsey <ke...@gmail.com>.
kentdorsey wrote:
>
>
> The questions:
>
> What is triggering the above select statement?
>
>
Looking at the JournalPersistenceAdapter source, every five minutes, a
ClockTicket invokes JournalPersistenceAdapter.checkpoint(false), which in
our case then invokes JDBCPersistenceAdapter.cleanup(), which eventually
invokes the above select statement via
DefaultJDBCAdapter.doGetExpiredMessages().
Also, looking at the JDBCPersistenceAdapatersource, every five minutes, a
ClockTicket invokes JDBCPersistenceAdapater.cleanup(), which also invokes
the above select statement via DefaultJDBCAdapter.doGetExpiredMessages().
There is a flag, isAutoCleanupExpiredMessages(), which affects the
execution.
New questions:
Do both adapters run the cleanup process every five minutes using
configuration defaults?
--
View this message in context: http://www.nabble.com/Persistent-messages-not-being-cleaned-up-for-non-durable-Topic-tf2242336.html#a6219674
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Persistent messages not being cleaned up for non-durable Topic
Posted by kentdorsey <ke...@gmail.com>.
Adrian Co wrote:
>
> FWIW, 3.x is pretty old. I would go for the 4.x code. :) Let us know how
> your tests go with the 4.x code.
>
Unfortunately, the project in question probably will not be able to upgrade
to 4.x anytime soon, due to platform limitations.
I am not sure that persisting the messages in question gains any ability to
recover them, due to the implementation strategy. For now, persistence of
these types of messages has been disabled in the application. The same
brokers are used for queues, so the underlying configuation needs to remain
stable for the moment.
I will schedule some time this week to perform additional tests on 3.x and
4.x and post the results, and look into the effects of setting the session
to be transacted on the client application.
If the configuration is not properly supported, and 3.x is still available,
a patch should be issued. I may be able to contribute towards that effort,
depending on how my schedule looks over the next week or two.
Kent
--
View this message in context: http://www.nabble.com/Persistent-messages-not-being-cleaned-up-for-non-durable-Topic-tf2242336.html#a6258751
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Persistent messages not being cleaned up for non-durable Topic
Posted by Adrian Co <ac...@exist.com>.
FWIW, 3.x is pretty old. I would go for the 4.x code. :) Let us know how
your tests go with the 4.x code.
kentdorsey wrote:
> The ActiveMQ 3.2.1 handling of persistent messages delivered to non-durable
> topic subscriptions appears to incorrectly persist messages that have been
> successfully received.
>
> I set up a test case with a single message broker configured to use cached,
> journaled, jdbc persistence against an HSQLDB database to do some local
> testing.
>
> Modified the ProducerTool to set the delivery mode to
> DeliveryMode.PERSISTENT for non-durable as well as durable topic messages to
> match the configuration on the project in question.
>
> Ran two ConsumerTool instances as non-durable subscribers to
> dynamicTopics/bogus. Ran a ProducerTool to send messages. Messages were
> sent and received correctly.
>
> After several minutes, the messages were STILL persisted to the database,
> due to the 5 minute period checkpoint() call in the JDBC adapter.
>
> My hunch is that the discrepancy between the logic used to call
> DurableTopicMessageContainerManager in the DefaultBroker and the logic used
> to acknowledge a message in ActiveMQMessageConsumer and ActiveMQSession may
> be the root cause of the problem. Setting the Session to be transcated may
> solve the problem, not sure. Messages are persisted (correctly) when they
> are marked with DeliveryMode.PERSISTENT, but no acknowledge is ever sent,
> because the topic is not durable, among other things.
>
> Message.DEFAULT_DELIVERY_MODE is DeliveryMode.PERSISTENT, so most users new
> to ActiveMQ andJMS experience will be sending persistent messages.
>
> Have not tested this in 4.x.
>
> Kent
>
>
> kentdorsey wrote:
>
>> A project that I am working on is seeing a massive build-up (over 500,000)
>> persistent messages for aTopic with non-durable subscribers.
>>
>> The configuration:
>>
>> ActiveMQ 3.2.1
>> There is a network of brokers: A and B connected over a reliable protocol
>> network channel.
>> The brokers are being accessed as a cluster using a client connection URI
>> with the reliable protocol to support failover.
>> The topics in question have non-durable subscribers.
>> The messages are being published with defaults, i.e.
>> Message.DEFAULT_DELIVERY_MODE (persistent), Message.DEFAULT_PRIORITY,
>> Message.DEFAULT_TIME_TO_LIVE (no expiration).
>> The session acknowledgment mode is Session.AUTO_ACKNOWLEDGE.
>> The client behavior does not call acknowledge() on the message, since
>> Session.AUTO_ACKNOWLEDGE was specified.
>>
>> The problems:
>>
>> A slowdown in database performance :-)
>>
>> Over 500,000 messages have built up in the ACTIVEMQ_MSGS table over
>> several months.
>>
>> There are no entries in ACTIVEMS_ACKS. (which is a good thing)
>>
>> The following select statement is resulting in hundreds of thousands of
>> "physical reads" (per the DBA) each hour:
>>
>> SELECT ID, CONTAINER, MSG_ID, SENT_TO_DEADLETTER FROM ACTIVEMQ_MSGS
>> WHERE (EXPIRATION <> 0 AND EXPIRATION<:1)
>>
>> There is no index on the EXPIRATION column. There is a JIRA issue where
>> this has been corrected. The index will be added to the system in
>> question.
>>
>> The questions:
>>
>> What is triggering the above select statement?
>>
>> I have read several reports on the forums of persistent messages not being
>> deleted for non-durable topics. Is there possibly a defect related to the
>> above configuration?
>>
>> Thanks,
>> Kent Dorsey
>>
>>
>>
>
>
Re: Persistent messages not being cleaned up for non-durable Topic
Posted by kentdorsey <ke...@gmail.com>.
The ActiveMQ 3.2.1 handling of persistent messages delivered to non-durable
topic subscriptions appears to incorrectly persist messages that have been
successfully received.
I set up a test case with a single message broker configured to use cached,
journaled, jdbc persistence against an HSQLDB database to do some local
testing.
Modified the ProducerTool to set the delivery mode to
DeliveryMode.PERSISTENT for non-durable as well as durable topic messages to
match the configuration on the project in question.
Ran two ConsumerTool instances as non-durable subscribers to
dynamicTopics/bogus. Ran a ProducerTool to send messages. Messages were
sent and received correctly.
After several minutes, the messages were STILL persisted to the database,
due to the 5 minute period checkpoint() call in the JDBC adapter.
My hunch is that the discrepancy between the logic used to call
DurableTopicMessageContainerManager in the DefaultBroker and the logic used
to acknowledge a message in ActiveMQMessageConsumer and ActiveMQSession may
be the root cause of the problem. Setting the Session to be transcated may
solve the problem, not sure. Messages are persisted (correctly) when they
are marked with DeliveryMode.PERSISTENT, but no acknowledge is ever sent,
because the topic is not durable, among other things.
Message.DEFAULT_DELIVERY_MODE is DeliveryMode.PERSISTENT, so most users new
to ActiveMQ andJMS experience will be sending persistent messages.
Have not tested this in 4.x.
Kent
kentdorsey wrote:
>
> A project that I am working on is seeing a massive build-up (over 500,000)
> persistent messages for aTopic with non-durable subscribers.
>
> The configuration:
>
> ActiveMQ 3.2.1
> There is a network of brokers: A and B connected over a reliable protocol
> network channel.
> The brokers are being accessed as a cluster using a client connection URI
> with the reliable protocol to support failover.
> The topics in question have non-durable subscribers.
> The messages are being published with defaults, i.e.
> Message.DEFAULT_DELIVERY_MODE (persistent), Message.DEFAULT_PRIORITY,
> Message.DEFAULT_TIME_TO_LIVE (no expiration).
> The session acknowledgment mode is Session.AUTO_ACKNOWLEDGE.
> The client behavior does not call acknowledge() on the message, since
> Session.AUTO_ACKNOWLEDGE was specified.
>
> The problems:
>
> A slowdown in database performance :-)
>
> Over 500,000 messages have built up in the ACTIVEMQ_MSGS table over
> several months.
>
> There are no entries in ACTIVEMS_ACKS. (which is a good thing)
>
> The following select statement is resulting in hundreds of thousands of
> "physical reads" (per the DBA) each hour:
>
> SELECT ID, CONTAINER, MSG_ID, SENT_TO_DEADLETTER FROM ACTIVEMQ_MSGS
> WHERE (EXPIRATION <> 0 AND EXPIRATION<:1)
>
> There is no index on the EXPIRATION column. There is a JIRA issue where
> this has been corrected. The index will be added to the system in
> question.
>
> The questions:
>
> What is triggering the above select statement?
>
> I have read several reports on the forums of persistent messages not being
> deleted for non-durable topics. Is there possibly a defect related to the
> above configuration?
>
> Thanks,
> Kent Dorsey
>
>
--
View this message in context: http://www.nabble.com/Persistent-messages-not-being-cleaned-up-for-non-durable-Topic-tf2242336.html#a6227139
Sent from the ActiveMQ - User forum at Nabble.com.