You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by frank_at_zynga <fr...@zynga.com> on 2009/01/23 00:39:56 UTC
Transacted Messages Not Committed
Hi,
We're just starting to phase in the use of AMQ 5.2.0 in a high volume
environment and I've run into some strange behavior with transacted
sessions. The basic architecture of the consumer is a java daemon that
spawns a configurable number of single threaded consumers that implement
MessageListener- each opens its own connection and transacted session. In
the consumer onMessage() method session.commit() is being called upon
successful processing of the message- and I've verified that it is actually
executed. The problem is that despite the message being successfully
processed and session.commit() executed the messages remain as pending in
the queue. If the consumer daemon is stopped and re-started these messages
are consumed again (definitely not good). Note that session.rollback() was
NOT called in this scenario, all the messages were processed successfully.
Also note that these are persistent messages and we are using the default
AMQ message store.
I've pored over the documentation, these forums, and google results and have
not come up with a diagnosis. I have observed and read up on the issue
where if you're using a prefetch size of > 0 then a session rollback will
block any additional consumption of messages by that session until the
original problematic message is committed or sent to the dead letter queue.
However, this is not the same behavior as none of the messages are rolled
back. For now I've changed the consumers to use client acknowledge and that
works, but we would really like to use transacted sessions if possible.
Thanks,
-Frank
--
View this message in context: http://www.nabble.com/Transacted-Messages-Not-Committed-tp21615994p21615994.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Transacted Messages Not Committed
Posted by frank_at_zynga <fr...@zynga.com>.
Just FYI for any forum users, Gary has correctly identified the issue as
being one with my code. I have an abstract class that the single threaded
consumers subclass and which handles the creation of the connection and the
session. In this class I had static variables for the connection and
session which resulted in the session being shared across threads, obviously
a no-no as sessions need to be in a single thread context. The simple
solution was to make these class instance variables and that solved the
problem.
Thanks Gary.
frank_at_zynga wrote:
>
> Hi Gary,
>
> I've entered this in JIRA under Test Cases. The title is "Transacted
> Messages not Committed".
>
> Thanks,
> -Frank
>
>
> Gary Tully wrote:
>>
>> Hi Frank,
>> it is great to have a test case. Could you create a jira issue[1] for
>> this and attach your tests case with the ASF license grant. In that
>> way we can use your test case in the activemq test suite to ensure
>> this issue stays fixed when it is resolved.
>>
>> Thanks,
>> Gary.
>>
>> [1] https://issues.apache.org/activemq/browse/AMQ
>>
>>
>
>
--
View this message in context: http://www.nabble.com/Transacted-Messages-Not-Committed-tp21615994p21715772.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Transacted Messages Not Committed
Posted by frank_at_zynga <fr...@zynga.com>.
Hi Gary,
I've entered this in JIRA under Test Cases. The title is "Transacted
Messages not Committed".
Thanks,
-Frank
Gary Tully wrote:
>
> Hi Frank,
> it is great to have a test case. Could you create a jira issue[1] for
> this and attach your tests case with the ASF license grant. In that
> way we can use your test case in the activemq test suite to ensure
> this issue stays fixed when it is resolved.
>
> Thanks,
> Gary.
>
> [1] https://issues.apache.org/activemq/browse/AMQ
>
>
--
View this message in context: http://www.nabble.com/Transacted-Messages-Not-Committed-tp21615994p21712119.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Transacted Messages Not Committed
Posted by Gary Tully <ga...@gmail.com>.
Hi Frank,
it is great to have a test case. Could you create a jira issue[1] for
this and attach your tests case with the ASF license grant. In that
way we can use your test case in the activemq test suite to ensure
this issue stays fixed when it is resolved.
Thanks,
Gary.
[1] https://issues.apache.org/activemq/browse/AMQ
2009/1/23 frank_at_zynga <fr...@zynga.com>:
>
> Hmmm, one of the files didn't get uploaded. Let me try again...
> http://www.nabble.com/file/p21632866/TestCase.java TestCase.java
>
> frank_at_zynga wrote:
>>
>> Hi Gary,
>>
>> I was able to create a test case and reproduce this behavior on a
>> consistent basis. The source code is attached and is actually nearly
>> identical to the code we're using in production. While running through
>> some test scenarios I was able to discover some additional information:
>>
>> - The problem *only* occurs when the number of consumer threads is greater
>> than 1. I found this out by accident.
>> - The problem occurs whether or not a correlation ID is used on the
>> messages. I wasn't sure if this mattered or not, but it doesn't.
>>
>> Fortunately, as a result of working with this test case I discovered a
>> work around to the problem, which is to not only call session.commit() on
>> successful message processing but to then call message.acknowledge() as
>> well. This works like a charm, but it was my understanding that calling
>> message.acknowledge() was not necessary when using transacted sessions.
>> Please correct me if I'm wrong. Also, I suspect this behavior might be
>> due to something funky I'm doing with my code, but it seems very
>> straightforward to me.
>>
>> Thanks,
>> -Frank http://www.nabble.com/file/p21632842/AMQWorkerTest.java
>> AMQWorkerTest.java
>> http://www.nabble.com/file/p21632842/AMQWorkerTest.java AMQWorkerTest.java
>>
>>
>> Gary Tully wrote:
>>>
>>> Hi Frank,
>>> do you think you can produce a test case for the behavior you describe.
>>> Thanks,
>>> Gary.
>>>
>>> 2009/1/22 frank_at_zynga <fr...@zynga.com>:
>>>>
>>>> Hi,
>>>>
>>>> We're just starting to phase in the use of AMQ 5.2.0 in a high volume
>>>> environment and I've run into some strange behavior with transacted
>>>> sessions. The basic architecture of the consumer is a java daemon that
>>>> spawns a configurable number of single threaded consumers that implement
>>>> MessageListener- each opens its own connection and transacted session.
>>>> In
>>>> the consumer onMessage() method session.commit() is being called upon
>>>> successful processing of the message- and I've verified that it is
>>>> actually
>>>> executed. The problem is that despite the message being successfully
>>>> processed and session.commit() executed the messages remain as pending
>>>> in
>>>> the queue. If the consumer daemon is stopped and re-started these
>>>> messages
>>>> are consumed again (definitely not good). Note that session.rollback()
>>>> was
>>>> NOT called in this scenario, all the messages were processed
>>>> successfully.
>>>> Also note that these are persistent messages and we are using the
>>>> default
>>>> AMQ message store.
>>>>
>>>> I've pored over the documentation, these forums, and google results and
>>>> have
>>>> not come up with a diagnosis. I have observed and read up on the issue
>>>> where if you're using a prefetch size of > 0 then a session rollback
>>>> will
>>>> block any additional consumption of messages by that session until the
>>>> original problematic message is committed or sent to the dead letter
>>>> queue.
>>>> However, this is not the same behavior as none of the messages are
>>>> rolled
>>>> back. For now I've changed the consumers to use client acknowledge and
>>>> that
>>>> works, but we would really like to use transacted sessions if possible.
>>>>
>>>> Thanks,
>>>> -Frank
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/Transacted-Messages-Not-Committed-tp21615994p21615994.html
>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> http://blog.garytully.com
>>>
>>> Open Source SOA
>>> http://FUSESource.com
>>>
>>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Transacted-Messages-Not-Committed-tp21615994p21632866.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
--
http://blog.garytully.com
Open Source SOA
http://FUSESource.com
Re: Transacted Messages Not Committed
Posted by frank_at_zynga <fr...@zynga.com>.
Hmmm, one of the files didn't get uploaded. Let me try again...
http://www.nabble.com/file/p21632866/TestCase.java TestCase.java
frank_at_zynga wrote:
>
> Hi Gary,
>
> I was able to create a test case and reproduce this behavior on a
> consistent basis. The source code is attached and is actually nearly
> identical to the code we're using in production. While running through
> some test scenarios I was able to discover some additional information:
>
> - The problem *only* occurs when the number of consumer threads is greater
> than 1. I found this out by accident.
> - The problem occurs whether or not a correlation ID is used on the
> messages. I wasn't sure if this mattered or not, but it doesn't.
>
> Fortunately, as a result of working with this test case I discovered a
> work around to the problem, which is to not only call session.commit() on
> successful message processing but to then call message.acknowledge() as
> well. This works like a charm, but it was my understanding that calling
> message.acknowledge() was not necessary when using transacted sessions.
> Please correct me if I'm wrong. Also, I suspect this behavior might be
> due to something funky I'm doing with my code, but it seems very
> straightforward to me.
>
> Thanks,
> -Frank http://www.nabble.com/file/p21632842/AMQWorkerTest.java
> AMQWorkerTest.java
> http://www.nabble.com/file/p21632842/AMQWorkerTest.java AMQWorkerTest.java
>
>
> Gary Tully wrote:
>>
>> Hi Frank,
>> do you think you can produce a test case for the behavior you describe.
>> Thanks,
>> Gary.
>>
>> 2009/1/22 frank_at_zynga <fr...@zynga.com>:
>>>
>>> Hi,
>>>
>>> We're just starting to phase in the use of AMQ 5.2.0 in a high volume
>>> environment and I've run into some strange behavior with transacted
>>> sessions. The basic architecture of the consumer is a java daemon that
>>> spawns a configurable number of single threaded consumers that implement
>>> MessageListener- each opens its own connection and transacted session.
>>> In
>>> the consumer onMessage() method session.commit() is being called upon
>>> successful processing of the message- and I've verified that it is
>>> actually
>>> executed. The problem is that despite the message being successfully
>>> processed and session.commit() executed the messages remain as pending
>>> in
>>> the queue. If the consumer daemon is stopped and re-started these
>>> messages
>>> are consumed again (definitely not good). Note that session.rollback()
>>> was
>>> NOT called in this scenario, all the messages were processed
>>> successfully.
>>> Also note that these are persistent messages and we are using the
>>> default
>>> AMQ message store.
>>>
>>> I've pored over the documentation, these forums, and google results and
>>> have
>>> not come up with a diagnosis. I have observed and read up on the issue
>>> where if you're using a prefetch size of > 0 then a session rollback
>>> will
>>> block any additional consumption of messages by that session until the
>>> original problematic message is committed or sent to the dead letter
>>> queue.
>>> However, this is not the same behavior as none of the messages are
>>> rolled
>>> back. For now I've changed the consumers to use client acknowledge and
>>> that
>>> works, but we would really like to use transacted sessions if possible.
>>>
>>> Thanks,
>>> -Frank
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Transacted-Messages-Not-Committed-tp21615994p21615994.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> http://blog.garytully.com
>>
>> Open Source SOA
>> http://FUSESource.com
>>
>>
>
>
--
View this message in context: http://www.nabble.com/Transacted-Messages-Not-Committed-tp21615994p21632866.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Transacted Messages Not Committed
Posted by frank_at_zynga <fr...@zynga.com>.
Hi Gary,
I was able to create a test case and reproduce this behavior on a consistent
basis. The source code is attached and is actually nearly identical to the
code we're using in production. While running through some test scenarios I
was able to discover some additional information:
- The problem *only* occurs when the number of consumer threads is greater
than 1. I found this out by accident.
- The problem occurs whether or not a correlation ID is used on the
messages. I wasn't sure if this mattered or not, but it doesn't.
Fortunately, as a result of working with this test case I discovered a work
around to the problem, which is to not only call session.commit() on
successful message processing but to then call message.acknowledge() as
well. This works like a charm, but it was my understanding that calling
message.acknowledge() was not necessary when using transacted sessions.
Please correct me if I'm wrong. Also, I suspect this behavior might be due
to something funky I'm doing with my code, but it seems very straightforward
to me.
Thanks,
-Frank http://www.nabble.com/file/p21632842/AMQWorkerTest.java
AMQWorkerTest.java http://www.nabble.com/file/p21632842/AMQWorkerTest.java
AMQWorkerTest.java
Gary Tully wrote:
>
> Hi Frank,
> do you think you can produce a test case for the behavior you describe.
> Thanks,
> Gary.
>
> 2009/1/22 frank_at_zynga <fr...@zynga.com>:
>>
>> Hi,
>>
>> We're just starting to phase in the use of AMQ 5.2.0 in a high volume
>> environment and I've run into some strange behavior with transacted
>> sessions. The basic architecture of the consumer is a java daemon that
>> spawns a configurable number of single threaded consumers that implement
>> MessageListener- each opens its own connection and transacted session.
>> In
>> the consumer onMessage() method session.commit() is being called upon
>> successful processing of the message- and I've verified that it is
>> actually
>> executed. The problem is that despite the message being successfully
>> processed and session.commit() executed the messages remain as pending in
>> the queue. If the consumer daemon is stopped and re-started these
>> messages
>> are consumed again (definitely not good). Note that session.rollback()
>> was
>> NOT called in this scenario, all the messages were processed
>> successfully.
>> Also note that these are persistent messages and we are using the default
>> AMQ message store.
>>
>> I've pored over the documentation, these forums, and google results and
>> have
>> not come up with a diagnosis. I have observed and read up on the issue
>> where if you're using a prefetch size of > 0 then a session rollback will
>> block any additional consumption of messages by that session until the
>> original problematic message is committed or sent to the dead letter
>> queue.
>> However, this is not the same behavior as none of the messages are rolled
>> back. For now I've changed the consumers to use client acknowledge and
>> that
>> works, but we would really like to use transacted sessions if possible.
>>
>> Thanks,
>> -Frank
>> --
>> View this message in context:
>> http://www.nabble.com/Transacted-Messages-Not-Committed-tp21615994p21615994.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> http://blog.garytully.com
>
> Open Source SOA
> http://FUSESource.com
>
>
--
View this message in context: http://www.nabble.com/Transacted-Messages-Not-Committed-tp21615994p21632842.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Transacted Messages Not Committed
Posted by Gary Tully <ga...@gmail.com>.
Hi Frank,
do you think you can produce a test case for the behavior you describe.
Thanks,
Gary.
2009/1/22 frank_at_zynga <fr...@zynga.com>:
>
> Hi,
>
> We're just starting to phase in the use of AMQ 5.2.0 in a high volume
> environment and I've run into some strange behavior with transacted
> sessions. The basic architecture of the consumer is a java daemon that
> spawns a configurable number of single threaded consumers that implement
> MessageListener- each opens its own connection and transacted session. In
> the consumer onMessage() method session.commit() is being called upon
> successful processing of the message- and I've verified that it is actually
> executed. The problem is that despite the message being successfully
> processed and session.commit() executed the messages remain as pending in
> the queue. If the consumer daemon is stopped and re-started these messages
> are consumed again (definitely not good). Note that session.rollback() was
> NOT called in this scenario, all the messages were processed successfully.
> Also note that these are persistent messages and we are using the default
> AMQ message store.
>
> I've pored over the documentation, these forums, and google results and have
> not come up with a diagnosis. I have observed and read up on the issue
> where if you're using a prefetch size of > 0 then a session rollback will
> block any additional consumption of messages by that session until the
> original problematic message is committed or sent to the dead letter queue.
> However, this is not the same behavior as none of the messages are rolled
> back. For now I've changed the consumers to use client acknowledge and that
> works, but we would really like to use transacted sessions if possible.
>
> Thanks,
> -Frank
> --
> View this message in context: http://www.nabble.com/Transacted-Messages-Not-Committed-tp21615994p21615994.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
--
http://blog.garytully.com
Open Source SOA
http://FUSESource.com