You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@activemq.apache.org by "Nevin Chen (JIRA)" <ji...@apache.org> on 2015/11/20 09:05:11 UTC

[jira] [Commented] (AMQ-5489) JMSExpiration not working correctly with LevelDB

    [ https://issues.apache.org/jira/browse/AMQ-5489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15015392#comment-15015392 ] 

Nevin Chen commented on AMQ-5489:
---------------------------------

I also met the same issue in AMQ 5.12.0. Here comes what I did to reproduce this issue and the proposal to fix it.

*How to Reproduce*
1. Configure AMQ to send the expired topic message to a queue DLQ and the storage is leveldb.
2. A JMS makes a durable subscription to a topic and disconnects after the subscription.
3. Another JMS client sends a message with expiration time to the topic in step 2.
4. After the expiry time, the message will be moved to DLQ. This can be monitored by the AMQ web console.
5. Restart AMQ. You will find that the message disappear from the AMQ web console. Try to consume the message from the DLQ, nothing is received.

*Cause Analysis*
1. KahaDB works well. It means the cause should be related to the leveldb mechanism. 
2. In my understanding, the JMS message is persistent to a log file and leveldb will maintain to a reference to the position where the JMS message is stored in the log. After a message expires, AMQ will copy the expired JMS message and the reference to the posistion is also copied, then send to DLQ. When AMQ restarts, DLQ will recover the message from the persistent storage. Since the DLQ message shares the same reference to the JMS message data, the message in DLQ also has the expiration time that is the same as the original message. The expiry scanner will detect the DLQ message expires and remove it. That's why the message is lost after restart.
3. Actually, the message to DLQ is not completely the same as the origial message. Because the expiration time will be reset to 0 and some more message properties will be added. The DLQ message should not reuse the same reference to message data. For more details, please refer to org.apache.activemq.broker.region.RegionBroker.sendToDeadLetterQueue(ConnectionContext context, MessageReference node, Subscription subscription, Throwable poisonCause) method.

*Fixed Proposal*
In org.apache.activemq.broker.region.RegionBroker.sendToDeadLetterQueue(ConnectionContext context, MessageReference node, Subscription subscription, Throwable poisonCause) method, after the message is copied, set the dataLocator to null (message.getMessageId().setDataLocator(null);) to force leveldb to save the new JMS message data and refer to a new position.

> JMSExpiration not working correctly with LevelDB
> ------------------------------------------------
>
>                 Key: AMQ-5489
>                 URL: https://issues.apache.org/jira/browse/AMQ-5489
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: activemq-leveldb-store
>    Affects Versions: 5.10.0
>         Environment: Solaris and MacOS X, Java 1.6, Camel 2.14 is used for sending messages to the AMQ.
>            Reporter: Johannes Pieringer
>         Attachments: JConsole_-_ActiveMQ.png, JConsole_-_Retry.png, activemq.xml, jms-expiration-test.zip
>
>
> I'd like to create a setup where messages with an JMSExpiration header expire in a queue named "Retry" and are then moved to the "ActiveMQ.DLQ". The messages should then be consumed from the ActiveMQ.DLQ. As a specialty, the messages are 5MB large.
> The messages do expire in "Retry" and are moved to the "ActiveMQ.DLQ". When the expire however, I instantly see twice the number of expired messages on the "ActiveMQ.DLQ" (see the attached pictures). The following two pictures (JConsole) are taken after the AMQ was started and all previous messages and statistics where deleted during startup. 
> The error happens with LevelDB and ReplicatedLevelDB. It does not occur if KahaDB is used. Furthermore the error does not occure if useCache is true and only a couple of messages are sent. It does occur with useCache is true if many messages are sent. It always happens if useCache is set to false.
> Endpoint URI and Parameters: activemq:queue:Retry?preserveMessageQos=true 
> We also set the JMSExpiration header to 60 seconds in the future. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)