Behavior when a message expires while it is processing

[Mike Chao <mi...@gmail.com>]

Hi,

I am trying to understand the behavior when a message expires while it is
being processed. I have the following flow setup in my test program.

queue.start -> sleepProcessor -> queue.end

The sleepProcessor takes the message from queue.start and sleeps for 5
seconds. I send a message to queue.start with a JMSExpiration of 1 second
from System.currentTimeMillis(). I have setup dead letter queues for each
queue named DLQ.queue.start and DLQ.queue.end.

The behavior I see is that 1 message ends up in DLQ.queue.start and another
message ends up in DLQ.queue.end.
How does 1 message become 2?

The test program can be found here with source
testJMSExpiration.zip
<http://activemq.2283324.n4.nabble.com/file/n4680241/testJMSExpiration.zip>  

I have included a maven pom.xml and the test program can be run with the
following command
mvn camel:run

OS: Linux 3.5.0 (Mint 14)
JVM: 1.6
ActiveMQ: 5.7.0
Camel: 2.8.2

Any insight would be greatly appreciated.
Thank you



--
View this message in context: http://activemq.2283324.n4.nabble.com/Behavior-when-a-message-expires-while-it-is-processing-tp4680241.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Behavior when a message expires while it is processing

[Mike Chao <mi...@gmail.com>]

To get the JUnit test to pass, I created a BrokerFilter that will filter out
duplicate messages based on the String property myMessageId.

testJMSExpirationWithFilterPlugin.zip
<http://activemq.2283324.n4.nabble.com/file/n4680373/testJMSExpirationWithFilterPlugin.zip>  

The behavior now is that only 1 message ends up the the DLQs. Which is
DLQ.queue.start



--
View this message in context: http://activemq.2283324.n4.nabble.com/Behavior-when-a-message-expires-while-it-is-processing-tp4680241p4680373.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Behavior when a message expires while it is processing

[Mike Chao <mi...@gmail.com>]

So I ran testJMSExpirationAsn1 and got the following NPE.

java.lang.NullPointerException
       at
com.test.TestJMSExpirationNoCamel$DLQEndQueueConsumer.run(TestJMSExpirationNoCamel.java:188)
       at java.lang.Thread.run(Thread.java:722)

I changed consumer.receive(3000) to consumer.receive(9000) to fix the NPE.
The fix is here  testJMSExpiration2.zip
<http://activemq.2283324.n4.nabble.com/file/n4680370/testJMSExpiration2.zip>  

The JUnit still fails because it only expects 1 message in the DLQs.

>From my understanding of ActiveMQ, which is limited, the correct behavior is
2 messages in the DLQs. Since they are separate messages ActiveMQ puts them
in the DLQs since they are both expired.  Or is my understanding incorrect?



--
View this message in context: http://activemq.2283324.n4.nabble.com/Behavior-when-a-message-expires-while-it-is-processing-tp4680241p4680370.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Behavior when a message expires while it is processing

[artnaseef <ar...@artnaseef.com>]

Here is an updated version of the test without camel, as a JUnit.  "mvn test"
does the trick.

testJMSExpirationAsn1.zip
<http://activemq.2283324.n4.nabble.com/file/n4680322/testJMSExpirationAsn1.zip>  

Note it's expecting only 1 DLQ message and fails when it gets 2.  I upgraded
the version of ActiveMQ in the test to 5.9.0, where it still fails.



--
View this message in context: http://activemq.2283324.n4.nabble.com/Behavior-when-a-message-expires-while-it-is-processing-tp4680241p4680322.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Behavior when a message expires while it is processing

[artnaseef <ar...@artnaseef.com>]

Based on this, http://en.wikipedia.org/wiki/Idempotence, the operation
as-described is definitely not idempotent.



--
View this message in context: http://activemq.2283324.n4.nabble.com/Behavior-when-a-message-expires-while-it-is-processing-tp4680241p4680319.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Behavior when a message expires while it is processing

[artnaseef <ar...@artnaseef.com>]

Hmm, my understanding of idempotent is that the same message can be processed
more than once without causing problems.  Keep in mind there are many
scenarios with JMS in which you can receive a duplicate message, such as the
broker delivers the message and the client processes it, but the
broker/client connection closes before the client's ack is received by the
broker.



--
View this message in context: http://activemq.2283324.n4.nabble.com/Behavior-when-a-message-expires-while-it-is-processing-tp4680241p4680317.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Behavior when a message expires while it is processing

[Mike Chao <mi...@gmail.com>]

The impact of having more than 1 message in the DLQ is that the process that
reads from the DLQ will need to keep track of which messages it has
processed. This introduces state and since there could be multiple processes
reading from the DLQ the state would have to be shared. We are trying to
avoid introducing state.

Camel has an idempotent consumer to keep track of which message the
processor has seen. 

I am not sure if ActiveMQ can be configured to check for duplicate messages
based on a custom header. I guess it is possible to write a custom
BrokerFilter.



--
View this message in context: http://activemq.2283324.n4.nabble.com/Behavior-when-a-message-expires-while-it-is-processing-tp4680241p4680314.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Behavior when a message expires while it is processing

[artnaseef <ar...@artnaseef.com>]

This is likely why the test isn't working as-expected; the following:



should be just this:



I believe it is necessary to set the TTL a second time when producing the
message again.  In the camel case, I believe camel is doing that for you. 
I'll try to get this working to verify.



--
View this message in context: http://activemq.2283324.n4.nabble.com/Behavior-when-a-message-expires-while-it-is-processing-tp4680241p4680310.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Behavior when a message expires while it is processing

[artnaseef <ar...@artnaseef.com>]

I don't think there's an easy way to only get one message on the DLQ without
changing the expiration handling to prevent expiration of inflight messages.

What is the impact of having more than one copy of the message in DLQs?



--
View this message in context: http://activemq.2283324.n4.nabble.com/Behavior-when-a-message-expires-while-it-is-processing-tp4680241p4680309.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Behavior when a message expires while it is processing

[Mike Chao <mi...@gmail.com>]

I think what you are saying is correct from the broker's perspective. 

I tried coding up an example without using camel and I do not see the same
behavior.

The example can be found here
http://s000.tinyupload.com/?file_id=62097383795604422204

I am not sure sure I coded up the example correctly. I am very new to
ActiveMQ and Camel.

Any ideas on how I can configure Camel and ActiveMQ to generate only 1
message? 
Maybe have the consumer send the ACK immediately upon receiving the message?



--
View this message in context: http://activemq.2283324.n4.nabble.com/Behavior-when-a-message-expires-while-it-is-processing-tp4680241p4680287.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Behavior when a message expires while it is processing

[artnaseef <ar...@artnaseef.com>]

Haven't look at the camel code yet, but don't think I need to at this point -
it appears to be working as-documented.

ActiveMQ is expiring inflight messages.  In other words, the following is
happening (from the broker's perspecive):

<ul>
<li> camel consumer for queue.start is started
<li> message produced to broker on queue.start with expiration set
<li> camel consumer receives message
<li> message expires
<li> broker puts message into the DLQ (not sure whether this is expected
operation)
<li> camel route produces message to queue.end
<li> consumer finally acknowledges the message (by a tranaction commit())
<li> message on queue.end expires and is routed to the DLQ
<ul>

Would you mind to take the time to trim down the example to a non-camel test
case which shows an inflight message doing to the DLQ?  Then either you or I
can create a jira entry for this (preferrably you will do it to maintain the
history).

Given that slow consumption is a Very Bad Thing for any JMS solution, and
that delivering an inflight message to a DLQ, from where it might incur
additional processing, is also a Bad Thing, the broker should not be
expiring inflight messages until they return to the broker for reprocessing
(i.e. client does not successfully ACK the message).



--
View this message in context: http://activemq.2283324.n4.nabble.com/Behavior-when-a-message-expires-while-it-is-processing-tp4680241p4680263.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Behavior when a message expires while it is processing

[artnaseef <ar...@artnaseef.com>]

Curious example.  This actually appears to be more of a camel question than
an ActiveMQ question.

Wearing my ActiveMQ hat, I see this:

* message is produced to queue, queue.start, after manually setting
JMSExpiration
* message is consumed from queue in a transaction by camel
* the camel exchange goes into a sleep() processor
* the message is finally put in queue, queue.end

Normally, JMSExpiration is ignored by ActiveMQ, but I would need to see if
Camel is using magic to overcome that.  JMSExpiration is a provider-internal
header which it sets; the normal way to set an expiration is to set it on
the MessageProducer.  Looking at camel's JMS component, though, it seems to
use that header itself.

I suspect the DLQ.queue.end message is happening because camel is continuing
to apply that header on the message outbound to queue.end, and ActiveMQ has
nothing to consume queue.end, so that message times out.  However, that
doesn't explain the queue.start message going to the DLQ.

Let me look into camel a little to confirm how it operates with
JMSExpiration.



--
View this message in context: http://activemq.2283324.n4.nabble.com/Behavior-when-a-message-expires-while-it-is-processing-tp4680241p4680262.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.