You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by David Sitsky <si...@nuix.com> on 2007/10/03 09:50:17 UTC
Duplicate message delivered to consumers on queue with kaha persistence
I have discovered an issue with the latest activemq 5.0 (revision
581510) which seems to have been present for some time, at least for the
last month.
I have an application using transactions, and kaha persistence, and two
consumers feeding off a queue.
I sometimes see the same message being sent to two consumers. All
transactions were committed (no rollbacks), and as you can see by the
debug dump, the message has the same message-ID, but was delivered to
two separate consumers, running on separate threads, using sync receives
with a prefetch of 0, with separate sessions of course.
Is this a known problem? It doesn't always happen - seems to be a race
condition. If I turn off persistence, I don't seem to see it.
Here is the message being delivered to consumer 1:
2007-10-03 17:21:37,310 [Worker 060bacd87f4a40a4867113e2e3e553dd] 85475
DEBUG org.apache.activemq.ActiveMQMessageConsumer -
ID:tamarama-62436-1191396093006-2:2:3:2 received message:
MessageDispatch {commandId = 0, responseRequired = false, consumerId =
ID:tamarama-62436-1191396093006-2:2:3:2, destination =
queue://worker-items, message = ActiveMQObjectMessage {commandId = 50,
responseRequired = false, messageId =
ID:tamarama-62436-1191396093006-2:2:1:3:1, originalDestination = null,
originalTransactionId = null, producerId =
ID:tamarama-62436-1191396093006-2:2:1:3, destination =
queue://worker-items, transactionId =
TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0, timestamp =
1191396097286, arrival = 0, brokerInTime = 1191396097290, brokerOutTime
= 1191396097303, correlationId = null, replyTo = null, persistent =
true, type = null, priority = 4, groupID = null, groupSequence = 0,
targetConsumerId = null, compressed = false, userID = null, content =
org.apache.activemq.util.ByteSequence@fa8cbc, marshalledProperties =
null, dataStructure = null, redeliveryCounter = 0, size = 0, properties
= null, readOnlyProperties = true, readOnlyBody = true, droppable =
false}, redeliveryCounter = 0}
Here is the same message (check the message ID) being delivered to
consumer 2:
2007-10-03 17:21:37,422 [Worker 2c3169d1726b4781a423a239f65c7bd1] 85587
DEBUG org.apache.activemq.ActiveMQMessageConsumer -
ID:tamarama-62436-1191396093006-2:2:1:2 received message:
MessageDispatch {commandId = 0, responseRequired = false, consumerId =
ID:tamarama-62436-1191396093006-2:2:1:2, destination =
queue://worker-items, message = ActiveMQObjectMessage {commandId = 50,
responseRequired = false, messageId =
ID:tamarama-62436-1191396093006-2:2:1:3:1, originalDestination = null,
originalTransactionId = null, producerId =
ID:tamarama-62436-1191396093006-2:2:1:3, destination =
queue://worker-items, transactionId =
TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0, timestamp =
1191396097286, arrival = 0, brokerInTime = 1191396097290, brokerOutTime
= 1191396097417, correlationId = null, replyTo = null, persistent =
true, type = null, priority = 4, groupID = null, groupSequence = 0,
targetConsumerId = null, compressed = false, userID = null, content =
org.apache.activemq.util.ByteSequence@1a881cc, marshalledProperties =
null, dataStructure = null, redeliveryCounter = 0, size = 0, properties
= null, readOnlyProperties = true, readOnlyBody = true, droppable =
false}, redeliveryCounter = 0}
I'll need to do more investigations tomorrow, but if anybody has any
ideas, I'd appreciate some insights.
--
Cheers,
David
Nuix Pty Ltd
Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 9280 0699
Web: http://www.nuix.com Fax: +61 2 9212 6902
Re: Duplicate message delivered to consumers on queue with kaha persistence
Posted by Rob Davies <ra...@gmail.com>.
AMQ Store is the combination of the journal (which itself now uses
Kaha) and Kaha to store message references
cheers,
Rob
http://rajdavies.blogspot.com/
On Oct 4, 2007, at 8:04 PM, Daryl Richter wrote:
> On Oct 3, 2007, at 7:30 AM, Rob Davies wrote:
>
>> Do you get the same using AMQ store (the default in 5.0) - its the
>> prefered store over kaha in ActiveMQ 5.0
>
>
> Hmm. When I start a very recent (revision 581630) build of
> ActiveMQ with completely default config I get (among others) the
> following lines in my log:
>
> INFO KahaStore - Kaha Store using data
> directory /usr/local/apache/activemq-5.0-581630/data/localhost/kr-
> store/state
> INFO AMQPersistenceAdapter - Active data files: []
> WARN AMQPersistenceAdapter - The ReferenceStore is not
> valid - recovering ...
> INFO KahaStore - Kaha Store successfully
> deleted data directory /usr/local/apache/activemq-5.0-581630/data/
> localhost/kr-store/data
> INFO AMQPersistenceAdapter - Journal Recovery Started
> from: DataManager:(data-)
> INFO AMQPersistenceAdapter - Recovered 0 operations from
> redo log in 0.0080 seconds.
> INFO AMQPersistenceAdapter - Finished recovering the
> ReferenceStore
> INFO KahaStore - Kaha Store using data
> directory /usr/local/apache/activemq-5.0-581630/data/localhost/kr-
> store/data
>
> So, which do I have? AMQStore or Kaha?
>
>
>>
>>
>>
>> cheers,
>>
>> Rob
>>
>> http://rajdavies.blogspot.com/
>>
>>
>>
>> On Oct 3, 2007, at 8:50 AM, David Sitsky wrote:
>>
>>> I have discovered an issue with the latest activemq 5.0 (revision
>>> 581510) which seems to have been present for some time, at least
>>> for the last month.
>>>
>>> I have an application using transactions, and kaha persistence,
>>> and two consumers feeding off a queue.
>>>
>>> I sometimes see the same message being sent to two consumers.
>>> All transactions were committed (no rollbacks), and as you can
>>> see by the debug dump, the message has the same message-ID, but
>>> was delivered to two separate consumers, running on separate
>>> threads, using sync receives with a prefetch of 0, with separate
>>> sessions of course.
>>>
>>> Is this a known problem? It doesn't always happen - seems to be
>>> a race condition. If I turn off persistence, I don't seem to see
>>> it.
>>>
>>> Here is the message being delivered to consumer 1:
>>>
>>> 2007-10-03 17:21:37,310 [Worker 060bacd87f4a40a4867113e2e3e553dd]
>>> 85475 DEBUG org.apache.activemq.ActiveMQMessageConsumer -
>>> ID:tamarama-62436-1191396093006-2:2:3:2 received message:
>>> MessageDispatch {commandId = 0, responseRequired = false,
>>> consumerId = ID:tamarama-62436-1191396093006-2:2:3:2, destination
>>> = queue://worker-items, message = ActiveMQObjectMessage
>>> {commandId = 50, responseRequired = false, messageId =
>>> ID:tamarama-62436-1191396093006-2:2:1:3:1, originalDestination =
>>> null, originalTransactionId = null, producerId =
>>> ID:tamarama-62436-1191396093006-2:2:1:3, destination = queue://
>>> worker-items, transactionId =
>>> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0,
>>> timestamp = 1191396097286, arrival = 0, brokerInTime =
>>> 1191396097290, brokerOutTime = 1191396097303, correlationId =
>>> null, replyTo = null, persistent = true, type = null, priority =
>>> 4, groupID = null, groupSequence = 0, targetConsumerId = null,
>>> compressed = false, userID = null, content =
>>> org.apache.activemq.util.ByteSequence@fa8cbc,
>>> marshalledProperties = null, dataStructure = null,
>>> redeliveryCounter = 0, size = 0, properties = null,
>>> readOnlyProperties = true, readOnlyBody = true, droppable =
>>> false}, redeliveryCounter = 0}
>>>
>>> Here is the same message (check the message ID) being delivered
>>> to consumer 2:
>>>
>>> 2007-10-03 17:21:37,422 [Worker 2c3169d1726b4781a423a239f65c7bd1]
>>> 85587 DEBUG org.apache.activemq.ActiveMQMessageConsumer -
>>> ID:tamarama-62436-1191396093006-2:2:1:2 received message:
>>> MessageDispatch {commandId = 0, responseRequired = false,
>>> consumerId = ID:tamarama-62436-1191396093006-2:2:1:2, destination
>>> = queue://worker-items, message = ActiveMQObjectMessage
>>> {commandId = 50, responseRequired = false, messageId =
>>> ID:tamarama-62436-1191396093006-2:2:1:3:1, originalDestination =
>>> null, originalTransactionId = null, producerId =
>>> ID:tamarama-62436-1191396093006-2:2:1:3, destination = queue://
>>> worker-items, transactionId =
>>> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0,
>>> timestamp = 1191396097286, arrival = 0, brokerInTime =
>>> 1191396097290, brokerOutTime = 1191396097417, correlationId =
>>> null, replyTo = null, persistent = true, type = null, priority =
>>> 4, groupID = null, groupSequence = 0, targetConsumerId = null,
>>> compressed = false, userID = null, content =
>>> org.apache.activemq.util.ByteSequence@1a881cc,
>>> marshalledProperties = null, dataStructure = null,
>>> redeliveryCounter = 0, size = 0, properties = null,
>>> readOnlyProperties = true, readOnlyBody = true, droppable =
>>> false}, redeliveryCounter = 0}
>>>
>>> I'll need to do more investigations tomorrow, but if anybody has
>>> any ideas, I'd appreciate some insights.
>>>
>>> --
>>> Cheers,
>>> David
>>>
>>> Nuix Pty Ltd
>>> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2
>>> 9280 0699
>>> Web: http://www.nuix.com Fax: +61 2
>>> 9212 6902
>>
>
> --
> Daryl
> http://itsallsemantics.com
>
> "I see this as coming down to mutual respect. I want to respect the
> others I communicate with enough to tell them my truth without
> reservation and I want to respect them enough to listen to their
> truth. I want to respect their good intentions enough to believe
> that we can work past our disagreements."
>
> -- Kent Beck, 2006
>
Re: Duplicate message delivered to consumers on queue with kaha persistence
Posted by Daryl Richter <ng...@comcast.net>.
On Oct 3, 2007, at 7:30 AM, Rob Davies wrote:
> Do you get the same using AMQ store (the default in 5.0) - its the
> prefered store over kaha in ActiveMQ 5.0
Hmm. When I start a very recent (revision 581630) build of ActiveMQ
with completely default config I get (among others) the following
lines in my log:
INFO KahaStore - Kaha Store using data
directory /usr/local/apache/activemq-5.0-581630/data/localhost/kr-
store/state
INFO AMQPersistenceAdapter - Active data files: []
WARN AMQPersistenceAdapter - The ReferenceStore is not
valid - recovering ...
INFO KahaStore - Kaha Store successfully
deleted data directory /usr/local/apache/activemq-5.0-581630/data/
localhost/kr-store/data
INFO AMQPersistenceAdapter - Journal Recovery Started from:
DataManager:(data-)
INFO AMQPersistenceAdapter - Recovered 0 operations from
redo log in 0.0080 seconds.
INFO AMQPersistenceAdapter - Finished recovering the
ReferenceStore
INFO KahaStore - Kaha Store using data
directory /usr/local/apache/activemq-5.0-581630/data/localhost/kr-
store/data
So, which do I have? AMQStore or Kaha?
>
>
>
> cheers,
>
> Rob
>
> http://rajdavies.blogspot.com/
>
>
>
> On Oct 3, 2007, at 8:50 AM, David Sitsky wrote:
>
>> I have discovered an issue with the latest activemq 5.0 (revision
>> 581510) which seems to have been present for some time, at least
>> for the last month.
>>
>> I have an application using transactions, and kaha persistence,
>> and two consumers feeding off a queue.
>>
>> I sometimes see the same message being sent to two consumers. All
>> transactions were committed (no rollbacks), and as you can see by
>> the debug dump, the message has the same message-ID, but was
>> delivered to two separate consumers, running on separate threads,
>> using sync receives with a prefetch of 0, with separate sessions
>> of course.
>>
>> Is this a known problem? It doesn't always happen - seems to be a
>> race condition. If I turn off persistence, I don't seem to see it.
>>
>> Here is the message being delivered to consumer 1:
>>
>> 2007-10-03 17:21:37,310 [Worker 060bacd87f4a40a4867113e2e3e553dd]
>> 85475 DEBUG org.apache.activemq.ActiveMQMessageConsumer -
>> ID:tamarama-62436-1191396093006-2:2:3:2 received message:
>> MessageDispatch {commandId = 0, responseRequired = false,
>> consumerId = ID:tamarama-62436-1191396093006-2:2:3:2, destination
>> = queue://worker-items, message = ActiveMQObjectMessage {commandId
>> = 50, responseRequired = false, messageId =
>> ID:tamarama-62436-1191396093006-2:2:1:3:1, originalDestination =
>> null, originalTransactionId = null, producerId =
>> ID:tamarama-62436-1191396093006-2:2:1:3, destination = queue://
>> worker-items, transactionId =
>> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0,
>> timestamp = 1191396097286, arrival = 0, brokerInTime =
>> 1191396097290, brokerOutTime = 1191396097303, correlationId =
>> null, replyTo = null, persistent = true, type = null, priority =
>> 4, groupID = null, groupSequence = 0, targetConsumerId = null,
>> compressed = false, userID = null, content =
>> org.apache.activemq.util.ByteSequence@fa8cbc, marshalledProperties
>> = null, dataStructure = null, redeliveryCounter = 0, size = 0,
>> properties = null, readOnlyProperties = true, readOnlyBody = true,
>> droppable = false}, redeliveryCounter = 0}
>>
>> Here is the same message (check the message ID) being delivered to
>> consumer 2:
>>
>> 2007-10-03 17:21:37,422 [Worker 2c3169d1726b4781a423a239f65c7bd1]
>> 85587 DEBUG org.apache.activemq.ActiveMQMessageConsumer -
>> ID:tamarama-62436-1191396093006-2:2:1:2 received message:
>> MessageDispatch {commandId = 0, responseRequired = false,
>> consumerId = ID:tamarama-62436-1191396093006-2:2:1:2, destination
>> = queue://worker-items, message = ActiveMQObjectMessage {commandId
>> = 50, responseRequired = false, messageId =
>> ID:tamarama-62436-1191396093006-2:2:1:3:1, originalDestination =
>> null, originalTransactionId = null, producerId =
>> ID:tamarama-62436-1191396093006-2:2:1:3, destination = queue://
>> worker-items, transactionId =
>> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0,
>> timestamp = 1191396097286, arrival = 0, brokerInTime =
>> 1191396097290, brokerOutTime = 1191396097417, correlationId =
>> null, replyTo = null, persistent = true, type = null, priority =
>> 4, groupID = null, groupSequence = 0, targetConsumerId = null,
>> compressed = false, userID = null, content =
>> org.apache.activemq.util.ByteSequence@1a881cc,
>> marshalledProperties = null, dataStructure = null,
>> redeliveryCounter = 0, size = 0, properties = null,
>> readOnlyProperties = true, readOnlyBody = true, droppable =
>> false}, redeliveryCounter = 0}
>>
>> I'll need to do more investigations tomorrow, but if anybody has
>> any ideas, I'd appreciate some insights.
>>
>> --
>> Cheers,
>> David
>>
>> Nuix Pty Ltd
>> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2
>> 9280 0699
>> Web: http://www.nuix.com Fax: +61 2
>> 9212 6902
>
--
Daryl
http://itsallsemantics.com
"I see this as coming down to mutual respect. I want to respect the
others I communicate with enough to tell them my truth without
reservation and I want to respect them enough to listen to their
truth. I want to respect their good intentions enough to believe that
we can work past our disagreements."
-- Kent Beck, 2006
Re: Duplicate message delivered to consumers on queue with AMQ persistence
Posted by David Sitsky <si...@nuix.com>.
Hi Rob,
I'm afraid I didn't get any time today, and my project has moved on to
not using JMS persistence for messaging. We are using custom
checkpointing instead, which is a bit more lightweight than full JMS
persistence.
I'd still like to see this issue fixed, so I'll try to find some time in
the next week or so to produce a unit test which will hopefully show the
issue.
Cheers,
David
David Sitsky wrote:
> Hi Rob,
>
> Sorry to hear that - I've also spent ages. FWIW, it happens regardless
> of the value for persistentIndex, I've used both true and false. Ditto
> with syncOnWrite, which I guess is not surprising.
>
> I'll try to modify my unit test further today to more closely mimic my
> real application to see if I can reproduce it.
>
> Cheers,
> David
>
> Rob Davies wrote:
>> Hi Dave,
>>
>> spent ages trying to reproduce this with no luck - anybody else had
>> luck reproducing this on the latest?
>>
>>
>> cheers,
>>
>> Rob
>>
>> http://rajdavies.blogspot.com/
>>
>>
>>
>> On Oct 9, 2007, at 6:28 AM, David Sitsky wrote:
>>
>>> Hi Rob,
>>>
>>> Looks like the original issue I reported with duplicate messages
>>> being dispatched has resurfaced again. I don't see it happening in
>>> the original situation where I saw it with kaha persistence, but now
>>> with different data loaded in my application.
>>>
>>> This time the issue occurs with AMQ persistence. I have tried to add
>>> as much detail as possible here:
>>> https://issues.apache.org/activemq/browse/AMQ-1452.
>>>
>>> Unfortunately, I have had no luck with reproducing this issue in a
>>> unit test... but I hope the message logs attached to the issue might
>>> provide you with some clues.
>>>
>>> Cheers,
>>> David
>>>
>>> David Sitsky wrote:
>>>> Hi Rob,
>>>> Yes - I haven't seen the issue in my application since updating.
>>>> Many thanks again.
>>>> Cheers,
>>>> David
>>>> Rob Davies wrote:
>>>>> Hi David,
>>>>>
>>>>> I had a test case that periodically could reproduce this - it now
>>>>> can't after ensuring references are always freshed in the Kaha
>>>>> reference store part of AMQ Store. Is your application now fixed?
>>>>>
>>>>> thanks,
>>>>>
>>>>> Rob
>>>>> On Oct 8, 2007, at 2:48 AM, David Sitsky wrote:
>>>>>
>>>>>> Hi Rob,
>>>>>>
>>>>>> Looks like you have fixed this issue - many thanks.
>>>>>>
>>>>>> Cheers,
>>>>>> David
>>>>>>
>>>>>> Rob Davies wrote:
>>>>>>> Hi David,
>>>>>>> I've had a look through the log you've attached to AMQ-1445 - but
>>>>>>> I think i'll need some more information to successfully resolve
>>>>>>> whats going on. Obviously there's an attempt to retrieve a
>>>>>>> message that's already been deleted - I just need to find out
>>>>>>> why ;)
>>>>>>> Could I trouble you to try and devise a junit test case to
>>>>>>> reproduce?
>>>>>>> thanks,
>>>>>>> Rob
>>>>>>> On Oct 5, 2007, at 7:37 AM, David Sitsky wrote:
>>>>>>>> Hi Rob,
>>>>>>>>
>>>>>>>> I think I spoke too soon. While using AMQ store fixed the
>>>>>>>> initial issue I was dealing with, it seems to have other issues
>>>>>>>> in my application, where I see the following message before
>>>>>>>> everything hangs:
>>>>>>>>
>>>>>>>> RecoveryListenerAdapter - Message id ID... could not be
>>>>>>>> recovered from the data store!
>>>>>>>>
>>>>>>>> I created a JIRA record which contains the detailed debug
>>>>>>>> message logs:
>>>>>>>>
>>>>>>>> https://issues.apache.org/activemq/browse/AMQ-1445
>>>>>>>>
>>>>>>>> FWIW, JDBC persistence works fine. Any ideas what might be wrong?
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> David
>>>>>>>>
>>>>>>>> David Sitsky wrote:
>>>>>>>>> Hi Rob,
>>>>>>>>> Looks like the AMQ store works correctly - thanks for the advice.
>>>>>>>>> Cheers,
>>>>>>>>> David
>>>>>>>>> Rob Davies wrote:
>>>>>>>>>> Do you get the same using AMQ store (the default in 5.0) - its
>>>>>>>>>> the prefered store over kaha in ActiveMQ 5.0
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> cheers,
>>>>>>>>>>
>>>>>>>>>> Rob
>>>>>>>>>>
>>>>>>>>>> http://rajdavies.blogspot.com/
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Oct 3, 2007, at 8:50 AM, David Sitsky wrote:
>>>>>>>>>>
>>>>>>>>>>> I have discovered an issue with the latest activemq 5.0
>>>>>>>>>>> (revision 581510) which seems to have been present for some
>>>>>>>>>>> time, at least for the last month.
>>>>>>>>>>>
>>>>>>>>>>> I have an application using transactions, and kaha
>>>>>>>>>>> persistence, and two consumers feeding off a queue.
>>>>>>>>>>>
>>>>>>>>>>> I sometimes see the same message being sent to two
>>>>>>>>>>> consumers. All transactions were committed (no rollbacks),
>>>>>>>>>>> and as you can see by the debug dump, the message has the
>>>>>>>>>>> same message-ID, but was delivered to two separate consumers,
>>>>>>>>>>> running on separate threads, using sync receives with a
>>>>>>>>>>> prefetch of 0, with separate sessions of course.
>>>>>>>>>>>
>>>>>>>>>>> Is this a known problem? It doesn't always happen - seems to
>>>>>>>>>>> be a race condition. If I turn off persistence, I don't seem
>>>>>>>>>>> to see it.
>>>>>>>>>>>
>>>>>>>>>>> Here is the message being delivered to consumer 1:
>>>>>>>>>>>
>>>>>>>>>>> 2007-10-03 17:21:37,310 [Worker
>>>>>>>>>>> 060bacd87f4a40a4867113e2e3e553dd] 85475 DEBUG
>>>>>>>>>>> org.apache.activemq.ActiveMQMessageConsumer -
>>>>>>>>>>> ID:tamarama-62436-1191396093006-2:2:3:2 received message:
>>>>>>>>>>> MessageDispatch {commandId = 0, responseRequired = false,
>>>>>>>>>>> consumerId = ID:tamarama-62436-1191396093006-2:2:3:2,
>>>>>>>>>>> destination = queue://worker-items, message =
>>>>>>>>>>> ActiveMQObjectMessage {commandId = 50, responseRequired =
>>>>>>>>>>> false, messageId = ID:tamarama-62436-1191396093006-2:2:1:3:1,
>>>>>>>>>>> originalDestination = null, originalTransactionId = null,
>>>>>>>>>>> producerId = ID:tamarama-62436-1191396093006-2:2:1:3,
>>>>>>>>>>> destination = queue://worker-items, transactionId =
>>>>>>>>>>> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0,
>>>>>>>>>>> timestamp = 1191396097286, arrival = 0, brokerInTime =
>>>>>>>>>>> 1191396097290, brokerOutTime = 1191396097303, correlationId =
>>>>>>>>>>> null, replyTo = null, persistent = true, type = null,
>>>>>>>>>>> priority = 4, groupID = null, groupSequence = 0,
>>>>>>>>>>> targetConsumerId = null, compressed = false, userID = null,
>>>>>>>>>>> content = org.apache.activemq.util.ByteSequence@fa8cbc,
>>>>>>>>>>> marshalledProperties = null, dataStructure = null,
>>>>>>>>>>> redeliveryCounter = 0, size = 0, properties = null,
>>>>>>>>>>> readOnlyProperties = true, readOnlyBody = true, droppable =
>>>>>>>>>>> false}, redeliveryCounter = 0}
>>>>>>>>>>>
>>>>>>>>>>> Here is the same message (check the message ID) being
>>>>>>>>>>> delivered to consumer 2:
>>>>>>>>>>>
>>>>>>>>>>> 2007-10-03 17:21:37,422 [Worker
>>>>>>>>>>> 2c3169d1726b4781a423a239f65c7bd1] 85587 DEBUG
>>>>>>>>>>> org.apache.activemq.ActiveMQMessageConsumer -
>>>>>>>>>>> ID:tamarama-62436-1191396093006-2:2:1:2 received message:
>>>>>>>>>>> MessageDispatch {commandId = 0, responseRequired = false,
>>>>>>>>>>> consumerId = ID:tamarama-62436-1191396093006-2:2:1:2,
>>>>>>>>>>> destination = queue://worker-items, message =
>>>>>>>>>>> ActiveMQObjectMessage {commandId = 50, responseRequired =
>>>>>>>>>>> false, messageId = ID:tamarama-62436-1191396093006-2:2:1:3:1,
>>>>>>>>>>> originalDestination = null, originalTransactionId = null,
>>>>>>>>>>> producerId = ID:tamarama-62436-1191396093006-2:2:1:3,
>>>>>>>>>>> destination = queue://worker-items, transactionId =
>>>>>>>>>>> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0,
>>>>>>>>>>> timestamp = 1191396097286, arrival = 0, brokerInTime =
>>>>>>>>>>> 1191396097290, brokerOutTime = 1191396097417, correlationId =
>>>>>>>>>>> null, replyTo = null, persistent = true, type = null,
>>>>>>>>>>> priority = 4, groupID = null, groupSequence = 0,
>>>>>>>>>>> targetConsumerId = null, compressed = false, userID = null,
>>>>>>>>>>> content = org.apache.activemq.util.ByteSequence@1a881cc,
>>>>>>>>>>> marshalledProperties = null, dataStructure = null,
>>>>>>>>>>> redeliveryCounter = 0, size = 0, properties = null,
>>>>>>>>>>> readOnlyProperties = true, readOnlyBody = true, droppable =
>>>>>>>>>>> false}, redeliveryCounter = 0}
>>>>>>>>>>>
>>>>>>>>>>> I'll need to do more investigations tomorrow, but if anybody
>>>>>>>>>>> has any ideas, I'd appreciate some insights.
>>>>>>>>>>>
Re: Duplicate message delivered to consumers on queue with AMQ persistence
Posted by David Sitsky <si...@nuix.com>.
Hi Rob,
Sorry to hear that - I've also spent ages. FWIW, it happens regardless
of the value for persistentIndex, I've used both true and false. Ditto
with syncOnWrite, which I guess is not surprising.
I'll try to modify my unit test further today to more closely mimic my
real application to see if I can reproduce it.
Cheers,
David
Rob Davies wrote:
> Hi Dave,
>
> spent ages trying to reproduce this with no luck - anybody else had luck
> reproducing this on the latest?
>
>
> cheers,
>
> Rob
>
> http://rajdavies.blogspot.com/
>
>
>
> On Oct 9, 2007, at 6:28 AM, David Sitsky wrote:
>
>> Hi Rob,
>>
>> Looks like the original issue I reported with duplicate messages being
>> dispatched has resurfaced again. I don't see it happening in the
>> original situation where I saw it with kaha persistence, but now with
>> different data loaded in my application.
>>
>> This time the issue occurs with AMQ persistence. I have tried to add
>> as much detail as possible here:
>> https://issues.apache.org/activemq/browse/AMQ-1452.
>>
>> Unfortunately, I have had no luck with reproducing this issue in a
>> unit test... but I hope the message logs attached to the issue might
>> provide you with some clues.
>>
>> Cheers,
>> David
>>
>> David Sitsky wrote:
>>> Hi Rob,
>>> Yes - I haven't seen the issue in my application since updating.
>>> Many thanks again.
>>> Cheers,
>>> David
>>> Rob Davies wrote:
>>>> Hi David,
>>>>
>>>> I had a test case that periodically could reproduce this - it now
>>>> can't after ensuring references are always freshed in the Kaha
>>>> reference store part of AMQ Store. Is your application now fixed?
>>>>
>>>> thanks,
>>>>
>>>> Rob
>>>> On Oct 8, 2007, at 2:48 AM, David Sitsky wrote:
>>>>
>>>>> Hi Rob,
>>>>>
>>>>> Looks like you have fixed this issue - many thanks.
>>>>>
>>>>> Cheers,
>>>>> David
>>>>>
>>>>> Rob Davies wrote:
>>>>>> Hi David,
>>>>>> I've had a look through the log you've attached to AMQ-1445 - but
>>>>>> I think i'll need some more information to successfully resolve
>>>>>> whats going on. Obviously there's an attempt to retrieve a message
>>>>>> that's already been deleted - I just need to find out why ;)
>>>>>> Could I trouble you to try and devise a junit test case to reproduce?
>>>>>> thanks,
>>>>>> Rob
>>>>>> On Oct 5, 2007, at 7:37 AM, David Sitsky wrote:
>>>>>>> Hi Rob,
>>>>>>>
>>>>>>> I think I spoke too soon. While using AMQ store fixed the
>>>>>>> initial issue I was dealing with, it seems to have other issues
>>>>>>> in my application, where I see the following message before
>>>>>>> everything hangs:
>>>>>>>
>>>>>>> RecoveryListenerAdapter - Message id ID... could not be recovered
>>>>>>> from the data store!
>>>>>>>
>>>>>>> I created a JIRA record which contains the detailed debug message
>>>>>>> logs:
>>>>>>>
>>>>>>> https://issues.apache.org/activemq/browse/AMQ-1445
>>>>>>>
>>>>>>> FWIW, JDBC persistence works fine. Any ideas what might be wrong?
>>>>>>>
>>>>>>> Cheers,
>>>>>>> David
>>>>>>>
>>>>>>> David Sitsky wrote:
>>>>>>>> Hi Rob,
>>>>>>>> Looks like the AMQ store works correctly - thanks for the advice.
>>>>>>>> Cheers,
>>>>>>>> David
>>>>>>>> Rob Davies wrote:
>>>>>>>>> Do you get the same using AMQ store (the default in 5.0) - its
>>>>>>>>> the prefered store over kaha in ActiveMQ 5.0
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> cheers,
>>>>>>>>>
>>>>>>>>> Rob
>>>>>>>>>
>>>>>>>>> http://rajdavies.blogspot.com/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Oct 3, 2007, at 8:50 AM, David Sitsky wrote:
>>>>>>>>>
>>>>>>>>>> I have discovered an issue with the latest activemq 5.0
>>>>>>>>>> (revision 581510) which seems to have been present for some
>>>>>>>>>> time, at least for the last month.
>>>>>>>>>>
>>>>>>>>>> I have an application using transactions, and kaha
>>>>>>>>>> persistence, and two consumers feeding off a queue.
>>>>>>>>>>
>>>>>>>>>> I sometimes see the same message being sent to two consumers.
>>>>>>>>>> All transactions were committed (no rollbacks), and as you can
>>>>>>>>>> see by the debug dump, the message has the same message-ID,
>>>>>>>>>> but was delivered to two separate consumers, running on
>>>>>>>>>> separate threads, using sync receives with a prefetch of 0,
>>>>>>>>>> with separate sessions of course.
>>>>>>>>>>
>>>>>>>>>> Is this a known problem? It doesn't always happen - seems to
>>>>>>>>>> be a race condition. If I turn off persistence, I don't seem
>>>>>>>>>> to see it.
>>>>>>>>>>
>>>>>>>>>> Here is the message being delivered to consumer 1:
>>>>>>>>>>
>>>>>>>>>> 2007-10-03 17:21:37,310 [Worker
>>>>>>>>>> 060bacd87f4a40a4867113e2e3e553dd] 85475 DEBUG
>>>>>>>>>> org.apache.activemq.ActiveMQMessageConsumer -
>>>>>>>>>> ID:tamarama-62436-1191396093006-2:2:3:2 received message:
>>>>>>>>>> MessageDispatch {commandId = 0, responseRequired = false,
>>>>>>>>>> consumerId = ID:tamarama-62436-1191396093006-2:2:3:2,
>>>>>>>>>> destination = queue://worker-items, message =
>>>>>>>>>> ActiveMQObjectMessage {commandId = 50, responseRequired =
>>>>>>>>>> false, messageId = ID:tamarama-62436-1191396093006-2:2:1:3:1,
>>>>>>>>>> originalDestination = null, originalTransactionId = null,
>>>>>>>>>> producerId = ID:tamarama-62436-1191396093006-2:2:1:3,
>>>>>>>>>> destination = queue://worker-items, transactionId =
>>>>>>>>>> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0,
>>>>>>>>>> timestamp = 1191396097286, arrival = 0, brokerInTime =
>>>>>>>>>> 1191396097290, brokerOutTime = 1191396097303, correlationId =
>>>>>>>>>> null, replyTo = null, persistent = true, type = null, priority
>>>>>>>>>> = 4, groupID = null, groupSequence = 0, targetConsumerId =
>>>>>>>>>> null, compressed = false, userID = null, content =
>>>>>>>>>> org.apache.activemq.util.ByteSequence@fa8cbc,
>>>>>>>>>> marshalledProperties = null, dataStructure = null,
>>>>>>>>>> redeliveryCounter = 0, size = 0, properties = null,
>>>>>>>>>> readOnlyProperties = true, readOnlyBody = true, droppable =
>>>>>>>>>> false}, redeliveryCounter = 0}
>>>>>>>>>>
>>>>>>>>>> Here is the same message (check the message ID) being
>>>>>>>>>> delivered to consumer 2:
>>>>>>>>>>
>>>>>>>>>> 2007-10-03 17:21:37,422 [Worker
>>>>>>>>>> 2c3169d1726b4781a423a239f65c7bd1] 85587 DEBUG
>>>>>>>>>> org.apache.activemq.ActiveMQMessageConsumer -
>>>>>>>>>> ID:tamarama-62436-1191396093006-2:2:1:2 received message:
>>>>>>>>>> MessageDispatch {commandId = 0, responseRequired = false,
>>>>>>>>>> consumerId = ID:tamarama-62436-1191396093006-2:2:1:2,
>>>>>>>>>> destination = queue://worker-items, message =
>>>>>>>>>> ActiveMQObjectMessage {commandId = 50, responseRequired =
>>>>>>>>>> false, messageId = ID:tamarama-62436-1191396093006-2:2:1:3:1,
>>>>>>>>>> originalDestination = null, originalTransactionId = null,
>>>>>>>>>> producerId = ID:tamarama-62436-1191396093006-2:2:1:3,
>>>>>>>>>> destination = queue://worker-items, transactionId =
>>>>>>>>>> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0,
>>>>>>>>>> timestamp = 1191396097286, arrival = 0, brokerInTime =
>>>>>>>>>> 1191396097290, brokerOutTime = 1191396097417, correlationId =
>>>>>>>>>> null, replyTo = null, persistent = true, type = null, priority
>>>>>>>>>> = 4, groupID = null, groupSequence = 0, targetConsumerId =
>>>>>>>>>> null, compressed = false, userID = null, content =
>>>>>>>>>> org.apache.activemq.util.ByteSequence@1a881cc,
>>>>>>>>>> marshalledProperties = null, dataStructure = null,
>>>>>>>>>> redeliveryCounter = 0, size = 0, properties = null,
>>>>>>>>>> readOnlyProperties = true, readOnlyBody = true, droppable =
>>>>>>>>>> false}, redeliveryCounter = 0}
>>>>>>>>>>
>>>>>>>>>> I'll need to do more investigations tomorrow, but if anybody
>>>>>>>>>> has any ideas, I'd appreciate some insights.
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Cheers,
>>>>>>>>>> David
>>>>>>>>>>
>>>>>>>>>> Nuix Pty Ltd
>>>>>>>>>> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2
>>>>>>>>>> 9280 0699
>>>>>>>>>> Web: http://www.nuix.com Fax: +61 2
>>>>>>>>>> 9212 6902
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Cheers,
>>>>>>> David
>>>>>>>
>>>>>>> Nuix Pty Ltd
>>>>>>> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2
>>>>>>> 9280 0699
>>>>>>> Web: http://www.nuix.com Fax: +61 2
>>>>>>> 9212 6902
>>>>>
>>>>>
>>>>> --
>>>>> Cheers,
>>>>> David
>>>>>
>>>>> Nuix Pty Ltd
>>>>> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 9280
>>>>> 0699
>>>>> Web: http://www.nuix.com Fax: +61 2 9212
>>>>> 6902
>>
>>
>> --
>> Cheers,
>> David
>>
>> Nuix Pty Ltd
>> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 9280 0699
>> Web: http://www.nuix.com Fax: +61 2 9212 6902
--
Cheers,
David
Nuix Pty Ltd
Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 9280 0699
Web: http://www.nuix.com Fax: +61 2 9212 6902
Re: Duplicate message delivered to consumers on queue with AMQ persistence
Posted by Rob Davies <ra...@gmail.com>.
Hi Dave,
spent ages trying to reproduce this with no luck - anybody else had
luck reproducing this on the latest?
cheers,
Rob
http://rajdavies.blogspot.com/
On Oct 9, 2007, at 6:28 AM, David Sitsky wrote:
> Hi Rob,
>
> Looks like the original issue I reported with duplicate messages
> being dispatched has resurfaced again. I don't see it happening in
> the original situation where I saw it with kaha persistence, but
> now with different data loaded in my application.
>
> This time the issue occurs with AMQ persistence. I have tried to
> add as much detail as possible here: https://issues.apache.org/
> activemq/browse/AMQ-1452.
>
> Unfortunately, I have had no luck with reproducing this issue in a
> unit test... but I hope the message logs attached to the issue
> might provide you with some clues.
>
> Cheers,
> David
>
> David Sitsky wrote:
>> Hi Rob,
>> Yes - I haven't seen the issue in my application since updating.
>> Many thanks again.
>> Cheers,
>> David
>> Rob Davies wrote:
>>> Hi David,
>>>
>>> I had a test case that periodically could reproduce this - it now
>>> can't after ensuring references are always freshed in the Kaha
>>> reference store part of AMQ Store. Is your application now fixed?
>>>
>>> thanks,
>>>
>>> Rob
>>> On Oct 8, 2007, at 2:48 AM, David Sitsky wrote:
>>>
>>>> Hi Rob,
>>>>
>>>> Looks like you have fixed this issue - many thanks.
>>>>
>>>> Cheers,
>>>> David
>>>>
>>>> Rob Davies wrote:
>>>>> Hi David,
>>>>> I've had a look through the log you've attached to AMQ-1445 -
>>>>> but I think i'll need some more information to successfully
>>>>> resolve whats going on. Obviously there's an attempt to
>>>>> retrieve a message that's already been deleted - I just need
>>>>> to find out why ;)
>>>>> Could I trouble you to try and devise a junit test case to
>>>>> reproduce?
>>>>> thanks,
>>>>> Rob
>>>>> On Oct 5, 2007, at 7:37 AM, David Sitsky wrote:
>>>>>> Hi Rob,
>>>>>>
>>>>>> I think I spoke too soon. While using AMQ store fixed the
>>>>>> initial issue I was dealing with, it seems to have other
>>>>>> issues in my application, where I see the following message
>>>>>> before everything hangs:
>>>>>>
>>>>>> RecoveryListenerAdapter - Message id ID... could not be
>>>>>> recovered from the data store!
>>>>>>
>>>>>> I created a JIRA record which contains the detailed debug
>>>>>> message logs:
>>>>>>
>>>>>> https://issues.apache.org/activemq/browse/AMQ-1445
>>>>>>
>>>>>> FWIW, JDBC persistence works fine. Any ideas what might be
>>>>>> wrong?
>>>>>>
>>>>>> Cheers,
>>>>>> David
>>>>>>
>>>>>> David Sitsky wrote:
>>>>>>> Hi Rob,
>>>>>>> Looks like the AMQ store works correctly - thanks for the
>>>>>>> advice.
>>>>>>> Cheers,
>>>>>>> David
>>>>>>> Rob Davies wrote:
>>>>>>>> Do you get the same using AMQ store (the default in 5.0) -
>>>>>>>> its the prefered store over kaha in ActiveMQ 5.0
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> cheers,
>>>>>>>>
>>>>>>>> Rob
>>>>>>>>
>>>>>>>> http://rajdavies.blogspot.com/
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Oct 3, 2007, at 8:50 AM, David Sitsky wrote:
>>>>>>>>
>>>>>>>>> I have discovered an issue with the latest activemq 5.0
>>>>>>>>> (revision 581510) which seems to have been present for some
>>>>>>>>> time, at least for the last month.
>>>>>>>>>
>>>>>>>>> I have an application using transactions, and kaha
>>>>>>>>> persistence, and two consumers feeding off a queue.
>>>>>>>>>
>>>>>>>>> I sometimes see the same message being sent to two
>>>>>>>>> consumers. All transactions were committed (no rollbacks),
>>>>>>>>> and as you can see by the debug dump, the message has the
>>>>>>>>> same message-ID, but was delivered to two separate
>>>>>>>>> consumers, running on separate threads, using sync receives
>>>>>>>>> with a prefetch of 0, with separate sessions of course.
>>>>>>>>>
>>>>>>>>> Is this a known problem? It doesn't always happen - seems
>>>>>>>>> to be a race condition. If I turn off persistence, I don't
>>>>>>>>> seem to see it.
>>>>>>>>>
>>>>>>>>> Here is the message being delivered to consumer 1:
>>>>>>>>>
>>>>>>>>> 2007-10-03 17:21:37,310 [Worker
>>>>>>>>> 060bacd87f4a40a4867113e2e3e553dd] 85475 DEBUG
>>>>>>>>> org.apache.activemq.ActiveMQMessageConsumer -
>>>>>>>>> ID:tamarama-62436-1191396093006-2:2:3:2 received message:
>>>>>>>>> MessageDispatch {commandId = 0, responseRequired = false,
>>>>>>>>> consumerId = ID:tamarama-62436-1191396093006-2:2:3:2,
>>>>>>>>> destination = queue://worker-items, message =
>>>>>>>>> ActiveMQObjectMessage {commandId = 50, responseRequired =
>>>>>>>>> false, messageId =
>>>>>>>>> ID:tamarama-62436-1191396093006-2:2:1:3:1,
>>>>>>>>> originalDestination = null, originalTransactionId = null,
>>>>>>>>> producerId = ID:tamarama-62436-1191396093006-2:2:1:3,
>>>>>>>>> destination = queue://worker-items, transactionId =
>>>>>>>>> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0,
>>>>>>>>> timestamp = 1191396097286, arrival = 0, brokerInTime =
>>>>>>>>> 1191396097290, brokerOutTime = 1191396097303, correlationId
>>>>>>>>> = null, replyTo = null, persistent = true, type = null,
>>>>>>>>> priority = 4, groupID = null, groupSequence = 0,
>>>>>>>>> targetConsumerId = null, compressed = false, userID = null,
>>>>>>>>> content = org.apache.activemq.util.ByteSequence@fa8cbc,
>>>>>>>>> marshalledProperties = null, dataStructure = null,
>>>>>>>>> redeliveryCounter = 0, size = 0, properties = null,
>>>>>>>>> readOnlyProperties = true, readOnlyBody = true, droppable =
>>>>>>>>> false}, redeliveryCounter = 0}
>>>>>>>>>
>>>>>>>>> Here is the same message (check the message ID) being
>>>>>>>>> delivered to consumer 2:
>>>>>>>>>
>>>>>>>>> 2007-10-03 17:21:37,422 [Worker
>>>>>>>>> 2c3169d1726b4781a423a239f65c7bd1] 85587 DEBUG
>>>>>>>>> org.apache.activemq.ActiveMQMessageConsumer -
>>>>>>>>> ID:tamarama-62436-1191396093006-2:2:1:2 received message:
>>>>>>>>> MessageDispatch {commandId = 0, responseRequired = false,
>>>>>>>>> consumerId = ID:tamarama-62436-1191396093006-2:2:1:2,
>>>>>>>>> destination = queue://worker-items, message =
>>>>>>>>> ActiveMQObjectMessage {commandId = 50, responseRequired =
>>>>>>>>> false, messageId =
>>>>>>>>> ID:tamarama-62436-1191396093006-2:2:1:3:1,
>>>>>>>>> originalDestination = null, originalTransactionId = null,
>>>>>>>>> producerId = ID:tamarama-62436-1191396093006-2:2:1:3,
>>>>>>>>> destination = queue://worker-items, transactionId =
>>>>>>>>> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0,
>>>>>>>>> timestamp = 1191396097286, arrival = 0, brokerInTime =
>>>>>>>>> 1191396097290, brokerOutTime = 1191396097417, correlationId
>>>>>>>>> = null, replyTo = null, persistent = true, type = null,
>>>>>>>>> priority = 4, groupID = null, groupSequence = 0,
>>>>>>>>> targetConsumerId = null, compressed = false, userID = null,
>>>>>>>>> content = org.apache.activemq.util.ByteSequence@1a881cc,
>>>>>>>>> marshalledProperties = null, dataStructure = null,
>>>>>>>>> redeliveryCounter = 0, size = 0, properties = null,
>>>>>>>>> readOnlyProperties = true, readOnlyBody = true, droppable =
>>>>>>>>> false}, redeliveryCounter = 0}
>>>>>>>>>
>>>>>>>>> I'll need to do more investigations tomorrow, but if
>>>>>>>>> anybody has any ideas, I'd appreciate some insights.
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Cheers,
>>>>>>>>> David
>>>>>>>>>
>>>>>>>>> Nuix Pty Ltd
>>>>>>>>> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph:
>>>>>>>>> +61 2 9280 0699
>>>>>>>>> Web: http://www.nuix.com Fax:
>>>>>>>>> +61 2 9212 6902
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Cheers,
>>>>>> David
>>>>>>
>>>>>> Nuix Pty Ltd
>>>>>> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2
>>>>>> 9280 0699
>>>>>> Web: http://www.nuix.com Fax: +61 2
>>>>>> 9212 6902
>>>>
>>>>
>>>> --
>>>> Cheers,
>>>> David
>>>>
>>>> Nuix Pty Ltd
>>>> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2
>>>> 9280 0699
>>>> Web: http://www.nuix.com Fax: +61 2
>>>> 9212 6902
>
>
> --
> Cheers,
> David
>
> Nuix Pty Ltd
> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 9280
> 0699
> Web: http://www.nuix.com Fax: +61 2 9212
> 6902
Re: Duplicate message delivered to consumers on queue with AMQ persistence
Posted by Rob Davies <ra...@gmail.com>.
Hi David,
are you trying this with the persistentIndex option?
thanks,
Rob
On Oct 9, 2007, at 6:28 AM, David Sitsky wrote:
> Hi Rob,
>
> Looks like the original issue I reported with duplicate messages
> being dispatched has resurfaced again. I don't see it happening in
> the original situation where I saw it with kaha persistence, but
> now with different data loaded in my application.
>
> This time the issue occurs with AMQ persistence. I have tried to
> add as much detail as possible here: https://issues.apache.org/
> activemq/browse/AMQ-1452.
>
> Unfortunately, I have had no luck with reproducing this issue in a
> unit test... but I hope the message logs attached to the issue
> might provide you with some clues.
>
> Cheers,
> David
>
> David Sitsky wrote:
>> Hi Rob,
>> Yes - I haven't seen the issue in my application since updating.
>> Many thanks again.
>> Cheers,
>> David
>> Rob Davies wrote:
>>> Hi David,
>>>
>>> I had a test case that periodically could reproduce this - it now
>>> can't after ensuring references are always freshed in the Kaha
>>> reference store part of AMQ Store. Is your application now fixed?
>>>
>>> thanks,
>>>
>>> Rob
>>> On Oct 8, 2007, at 2:48 AM, David Sitsky wrote:
>>>
>>>> Hi Rob,
>>>>
>>>> Looks like you have fixed this issue - many thanks.
>>>>
>>>> Cheers,
>>>> David
>>>>
>>>> Rob Davies wrote:
>>>>> Hi David,
>>>>> I've had a look through the log you've attached to AMQ-1445 -
>>>>> but I think i'll need some more information to successfully
>>>>> resolve whats going on. Obviously there's an attempt to
>>>>> retrieve a message that's already been deleted - I just need
>>>>> to find out why ;)
>>>>> Could I trouble you to try and devise a junit test case to
>>>>> reproduce?
>>>>> thanks,
>>>>> Rob
>>>>> On Oct 5, 2007, at 7:37 AM, David Sitsky wrote:
>>>>>> Hi Rob,
>>>>>>
>>>>>> I think I spoke too soon. While using AMQ store fixed the
>>>>>> initial issue I was dealing with, it seems to have other
>>>>>> issues in my application, where I see the following message
>>>>>> before everything hangs:
>>>>>>
>>>>>> RecoveryListenerAdapter - Message id ID... could not be
>>>>>> recovered from the data store!
>>>>>>
>>>>>> I created a JIRA record which contains the detailed debug
>>>>>> message logs:
>>>>>>
>>>>>> https://issues.apache.org/activemq/browse/AMQ-1445
>>>>>>
>>>>>> FWIW, JDBC persistence works fine. Any ideas what might be
>>>>>> wrong?
>>>>>>
>>>>>> Cheers,
>>>>>> David
>>>>>>
>>>>>> David Sitsky wrote:
>>>>>>> Hi Rob,
>>>>>>> Looks like the AMQ store works correctly - thanks for the
>>>>>>> advice.
>>>>>>> Cheers,
>>>>>>> David
>>>>>>> Rob Davies wrote:
>>>>>>>> Do you get the same using AMQ store (the default in 5.0) -
>>>>>>>> its the prefered store over kaha in ActiveMQ 5.0
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> cheers,
>>>>>>>>
>>>>>>>> Rob
>>>>>>>>
>>>>>>>> http://rajdavies.blogspot.com/
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Oct 3, 2007, at 8:50 AM, David Sitsky wrote:
>>>>>>>>
>>>>>>>>> I have discovered an issue with the latest activemq 5.0
>>>>>>>>> (revision 581510) which seems to have been present for some
>>>>>>>>> time, at least for the last month.
>>>>>>>>>
>>>>>>>>> I have an application using transactions, and kaha
>>>>>>>>> persistence, and two consumers feeding off a queue.
>>>>>>>>>
>>>>>>>>> I sometimes see the same message being sent to two
>>>>>>>>> consumers. All transactions were committed (no rollbacks),
>>>>>>>>> and as you can see by the debug dump, the message has the
>>>>>>>>> same message-ID, but was delivered to two separate
>>>>>>>>> consumers, running on separate threads, using sync receives
>>>>>>>>> with a prefetch of 0, with separate sessions of course.
>>>>>>>>>
>>>>>>>>> Is this a known problem? It doesn't always happen - seems
>>>>>>>>> to be a race condition. If I turn off persistence, I don't
>>>>>>>>> seem to see it.
>>>>>>>>>
>>>>>>>>> Here is the message being delivered to consumer 1:
>>>>>>>>>
>>>>>>>>> 2007-10-03 17:21:37,310 [Worker
>>>>>>>>> 060bacd87f4a40a4867113e2e3e553dd] 85475 DEBUG
>>>>>>>>> org.apache.activemq.ActiveMQMessageConsumer -
>>>>>>>>> ID:tamarama-62436-1191396093006-2:2:3:2 received message:
>>>>>>>>> MessageDispatch {commandId = 0, responseRequired = false,
>>>>>>>>> consumerId = ID:tamarama-62436-1191396093006-2:2:3:2,
>>>>>>>>> destination = queue://worker-items, message =
>>>>>>>>> ActiveMQObjectMessage {commandId = 50, responseRequired =
>>>>>>>>> false, messageId =
>>>>>>>>> ID:tamarama-62436-1191396093006-2:2:1:3:1,
>>>>>>>>> originalDestination = null, originalTransactionId = null,
>>>>>>>>> producerId = ID:tamarama-62436-1191396093006-2:2:1:3,
>>>>>>>>> destination = queue://worker-items, transactionId =
>>>>>>>>> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0,
>>>>>>>>> timestamp = 1191396097286, arrival = 0, brokerInTime =
>>>>>>>>> 1191396097290, brokerOutTime = 1191396097303, correlationId
>>>>>>>>> = null, replyTo = null, persistent = true, type = null,
>>>>>>>>> priority = 4, groupID = null, groupSequence = 0,
>>>>>>>>> targetConsumerId = null, compressed = false, userID = null,
>>>>>>>>> content = org.apache.activemq.util.ByteSequence@fa8cbc,
>>>>>>>>> marshalledProperties = null, dataStructure = null,
>>>>>>>>> redeliveryCounter = 0, size = 0, properties = null,
>>>>>>>>> readOnlyProperties = true, readOnlyBody = true, droppable =
>>>>>>>>> false}, redeliveryCounter = 0}
>>>>>>>>>
>>>>>>>>> Here is the same message (check the message ID) being
>>>>>>>>> delivered to consumer 2:
>>>>>>>>>
>>>>>>>>> 2007-10-03 17:21:37,422 [Worker
>>>>>>>>> 2c3169d1726b4781a423a239f65c7bd1] 85587 DEBUG
>>>>>>>>> org.apache.activemq.ActiveMQMessageConsumer -
>>>>>>>>> ID:tamarama-62436-1191396093006-2:2:1:2 received message:
>>>>>>>>> MessageDispatch {commandId = 0, responseRequired = false,
>>>>>>>>> consumerId = ID:tamarama-62436-1191396093006-2:2:1:2,
>>>>>>>>> destination = queue://worker-items, message =
>>>>>>>>> ActiveMQObjectMessage {commandId = 50, responseRequired =
>>>>>>>>> false, messageId =
>>>>>>>>> ID:tamarama-62436-1191396093006-2:2:1:3:1,
>>>>>>>>> originalDestination = null, originalTransactionId = null,
>>>>>>>>> producerId = ID:tamarama-62436-1191396093006-2:2:1:3,
>>>>>>>>> destination = queue://worker-items, transactionId =
>>>>>>>>> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0,
>>>>>>>>> timestamp = 1191396097286, arrival = 0, brokerInTime =
>>>>>>>>> 1191396097290, brokerOutTime = 1191396097417, correlationId
>>>>>>>>> = null, replyTo = null, persistent = true, type = null,
>>>>>>>>> priority = 4, groupID = null, groupSequence = 0,
>>>>>>>>> targetConsumerId = null, compressed = false, userID = null,
>>>>>>>>> content = org.apache.activemq.util.ByteSequence@1a881cc,
>>>>>>>>> marshalledProperties = null, dataStructure = null,
>>>>>>>>> redeliveryCounter = 0, size = 0, properties = null,
>>>>>>>>> readOnlyProperties = true, readOnlyBody = true, droppable =
>>>>>>>>> false}, redeliveryCounter = 0}
>>>>>>>>>
>>>>>>>>> I'll need to do more investigations tomorrow, but if
>>>>>>>>> anybody has any ideas, I'd appreciate some insights.
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Cheers,
>>>>>>>>> David
>>>>>>>>>
>>>>>>>>> Nuix Pty Ltd
>>>>>>>>> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph:
>>>>>>>>> +61 2 9280 0699
>>>>>>>>> Web: http://www.nuix.com Fax:
>>>>>>>>> +61 2 9212 6902
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Cheers,
>>>>>> David
>>>>>>
>>>>>> Nuix Pty Ltd
>>>>>> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2
>>>>>> 9280 0699
>>>>>> Web: http://www.nuix.com Fax: +61 2
>>>>>> 9212 6902
>>>>
>>>>
>>>> --
>>>> Cheers,
>>>> David
>>>>
>>>> Nuix Pty Ltd
>>>> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2
>>>> 9280 0699
>>>> Web: http://www.nuix.com Fax: +61 2
>>>> 9212 6902
>
>
> --
> Cheers,
> David
>
> Nuix Pty Ltd
> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 9280
> 0699
> Web: http://www.nuix.com Fax: +61 2 9212
> 6902
Duplicate message delivered to consumers on queue with AMQ persistence
Posted by David Sitsky <si...@nuix.com>.
Hi Rob,
Looks like the original issue I reported with duplicate messages being
dispatched has resurfaced again. I don't see it happening in the
original situation where I saw it with kaha persistence, but now with
different data loaded in my application.
This time the issue occurs with AMQ persistence. I have tried to add as
much detail as possible here:
https://issues.apache.org/activemq/browse/AMQ-1452.
Unfortunately, I have had no luck with reproducing this issue in a unit
test... but I hope the message logs attached to the issue might provide
you with some clues.
Cheers,
David
David Sitsky wrote:
> Hi Rob,
>
> Yes - I haven't seen the issue in my application since updating. Many
> thanks again.
>
> Cheers,
> David
>
> Rob Davies wrote:
>> Hi David,
>>
>> I had a test case that periodically could reproduce this - it now
>> can't after ensuring references are always freshed in the Kaha
>> reference store part of AMQ Store. Is your application now fixed?
>>
>> thanks,
>>
>> Rob
>> On Oct 8, 2007, at 2:48 AM, David Sitsky wrote:
>>
>>> Hi Rob,
>>>
>>> Looks like you have fixed this issue - many thanks.
>>>
>>> Cheers,
>>> David
>>>
>>> Rob Davies wrote:
>>>> Hi David,
>>>> I've had a look through the log you've attached to AMQ-1445 - but I
>>>> think i'll need some more information to successfully resolve whats
>>>> going on. Obviously there's an attempt to retrieve a message that's
>>>> already been deleted - I just need to find out why ;)
>>>> Could I trouble you to try and devise a junit test case to reproduce?
>>>> thanks,
>>>> Rob
>>>> On Oct 5, 2007, at 7:37 AM, David Sitsky wrote:
>>>>> Hi Rob,
>>>>>
>>>>> I think I spoke too soon. While using AMQ store fixed the initial
>>>>> issue I was dealing with, it seems to have other issues in my
>>>>> application, where I see the following message before everything
>>>>> hangs:
>>>>>
>>>>> RecoveryListenerAdapter - Message id ID... could not be recovered
>>>>> from the data store!
>>>>>
>>>>> I created a JIRA record which contains the detailed debug message
>>>>> logs:
>>>>>
>>>>> https://issues.apache.org/activemq/browse/AMQ-1445
>>>>>
>>>>> FWIW, JDBC persistence works fine. Any ideas what might be wrong?
>>>>>
>>>>> Cheers,
>>>>> David
>>>>>
>>>>> David Sitsky wrote:
>>>>>> Hi Rob,
>>>>>> Looks like the AMQ store works correctly - thanks for the advice.
>>>>>> Cheers,
>>>>>> David
>>>>>> Rob Davies wrote:
>>>>>>> Do you get the same using AMQ store (the default in 5.0) - its
>>>>>>> the prefered store over kaha in ActiveMQ 5.0
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> cheers,
>>>>>>>
>>>>>>> Rob
>>>>>>>
>>>>>>> http://rajdavies.blogspot.com/
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Oct 3, 2007, at 8:50 AM, David Sitsky wrote:
>>>>>>>
>>>>>>>> I have discovered an issue with the latest activemq 5.0
>>>>>>>> (revision 581510) which seems to have been present for some
>>>>>>>> time, at least for the last month.
>>>>>>>>
>>>>>>>> I have an application using transactions, and kaha persistence,
>>>>>>>> and two consumers feeding off a queue.
>>>>>>>>
>>>>>>>> I sometimes see the same message being sent to two consumers.
>>>>>>>> All transactions were committed (no rollbacks), and as you can
>>>>>>>> see by the debug dump, the message has the same message-ID, but
>>>>>>>> was delivered to two separate consumers, running on separate
>>>>>>>> threads, using sync receives with a prefetch of 0, with separate
>>>>>>>> sessions of course.
>>>>>>>>
>>>>>>>> Is this a known problem? It doesn't always happen - seems to be
>>>>>>>> a race condition. If I turn off persistence, I don't seem to
>>>>>>>> see it.
>>>>>>>>
>>>>>>>> Here is the message being delivered to consumer 1:
>>>>>>>>
>>>>>>>> 2007-10-03 17:21:37,310 [Worker
>>>>>>>> 060bacd87f4a40a4867113e2e3e553dd] 85475 DEBUG
>>>>>>>> org.apache.activemq.ActiveMQMessageConsumer -
>>>>>>>> ID:tamarama-62436-1191396093006-2:2:3:2 received message:
>>>>>>>> MessageDispatch {commandId = 0, responseRequired = false,
>>>>>>>> consumerId = ID:tamarama-62436-1191396093006-2:2:3:2,
>>>>>>>> destination = queue://worker-items, message =
>>>>>>>> ActiveMQObjectMessage {commandId = 50, responseRequired = false,
>>>>>>>> messageId = ID:tamarama-62436-1191396093006-2:2:1:3:1,
>>>>>>>> originalDestination = null, originalTransactionId = null,
>>>>>>>> producerId = ID:tamarama-62436-1191396093006-2:2:1:3,
>>>>>>>> destination = queue://worker-items, transactionId =
>>>>>>>> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0,
>>>>>>>> timestamp = 1191396097286, arrival = 0, brokerInTime =
>>>>>>>> 1191396097290, brokerOutTime = 1191396097303, correlationId =
>>>>>>>> null, replyTo = null, persistent = true, type = null, priority =
>>>>>>>> 4, groupID = null, groupSequence = 0, targetConsumerId = null,
>>>>>>>> compressed = false, userID = null, content =
>>>>>>>> org.apache.activemq.util.ByteSequence@fa8cbc,
>>>>>>>> marshalledProperties = null, dataStructure = null,
>>>>>>>> redeliveryCounter = 0, size = 0, properties = null,
>>>>>>>> readOnlyProperties = true, readOnlyBody = true, droppable =
>>>>>>>> false}, redeliveryCounter = 0}
>>>>>>>>
>>>>>>>> Here is the same message (check the message ID) being delivered
>>>>>>>> to consumer 2:
>>>>>>>>
>>>>>>>> 2007-10-03 17:21:37,422 [Worker
>>>>>>>> 2c3169d1726b4781a423a239f65c7bd1] 85587 DEBUG
>>>>>>>> org.apache.activemq.ActiveMQMessageConsumer -
>>>>>>>> ID:tamarama-62436-1191396093006-2:2:1:2 received message:
>>>>>>>> MessageDispatch {commandId = 0, responseRequired = false,
>>>>>>>> consumerId = ID:tamarama-62436-1191396093006-2:2:1:2,
>>>>>>>> destination = queue://worker-items, message =
>>>>>>>> ActiveMQObjectMessage {commandId = 50, responseRequired = false,
>>>>>>>> messageId = ID:tamarama-62436-1191396093006-2:2:1:3:1,
>>>>>>>> originalDestination = null, originalTransactionId = null,
>>>>>>>> producerId = ID:tamarama-62436-1191396093006-2:2:1:3,
>>>>>>>> destination = queue://worker-items, transactionId =
>>>>>>>> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0,
>>>>>>>> timestamp = 1191396097286, arrival = 0, brokerInTime =
>>>>>>>> 1191396097290, brokerOutTime = 1191396097417, correlationId =
>>>>>>>> null, replyTo = null, persistent = true, type = null, priority =
>>>>>>>> 4, groupID = null, groupSequence = 0, targetConsumerId = null,
>>>>>>>> compressed = false, userID = null, content =
>>>>>>>> org.apache.activemq.util.ByteSequence@1a881cc,
>>>>>>>> marshalledProperties = null, dataStructure = null,
>>>>>>>> redeliveryCounter = 0, size = 0, properties = null,
>>>>>>>> readOnlyProperties = true, readOnlyBody = true, droppable =
>>>>>>>> false}, redeliveryCounter = 0}
>>>>>>>>
>>>>>>>> I'll need to do more investigations tomorrow, but if anybody has
>>>>>>>> any ideas, I'd appreciate some insights.
>>>>>>>>
>>>>>>>> --
>>>>>>>> Cheers,
>>>>>>>> David
>>>>>>>>
>>>>>>>> Nuix Pty Ltd
>>>>>>>> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2
>>>>>>>> 9280 0699
>>>>>>>> Web: http://www.nuix.com Fax: +61 2
>>>>>>>> 9212 6902
>>>>>
>>>>>
>>>>> --
>>>>> Cheers,
>>>>> David
>>>>>
>>>>> Nuix Pty Ltd
>>>>> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 9280
>>>>> 0699
>>>>> Web: http://www.nuix.com Fax: +61 2 9212
>>>>> 6902
>>>
>>>
>>> --
>>> Cheers,
>>> David
>>>
>>> Nuix Pty Ltd
>>> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 9280 0699
>>> Web: http://www.nuix.com Fax: +61 2 9212 6902
>
>
--
Cheers,
David
Nuix Pty Ltd
Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 9280 0699
Web: http://www.nuix.com Fax: +61 2 9212 6902
Re: RecoveryListenerAdapter - Message id ID... could not be recovered
from the data store: WAS: [Duplicate message delivered to consumers on queue
with kaha persistence]
Posted by David Sitsky <si...@nuix.com>.
Hi Rob,
Yes - I haven't seen the issue in my application since updating. Many
thanks again.
Cheers,
David
Rob Davies wrote:
> Hi David,
>
> I had a test case that periodically could reproduce this - it now can't
> after ensuring references are always freshed in the Kaha reference store
> part of AMQ Store. Is your application now fixed?
>
> thanks,
>
> Rob
> On Oct 8, 2007, at 2:48 AM, David Sitsky wrote:
>
>> Hi Rob,
>>
>> Looks like you have fixed this issue - many thanks.
>>
>> Cheers,
>> David
>>
>> Rob Davies wrote:
>>> Hi David,
>>> I've had a look through the log you've attached to AMQ-1445 - but I
>>> think i'll need some more information to successfully resolve whats
>>> going on. Obviously there's an attempt to retrieve a message that's
>>> already been deleted - I just need to find out why ;)
>>> Could I trouble you to try and devise a junit test case to reproduce?
>>> thanks,
>>> Rob
>>> On Oct 5, 2007, at 7:37 AM, David Sitsky wrote:
>>>> Hi Rob,
>>>>
>>>> I think I spoke too soon. While using AMQ store fixed the initial
>>>> issue I was dealing with, it seems to have other issues in my
>>>> application, where I see the following message before everything hangs:
>>>>
>>>> RecoveryListenerAdapter - Message id ID... could not be recovered
>>>> from the data store!
>>>>
>>>> I created a JIRA record which contains the detailed debug message logs:
>>>>
>>>> https://issues.apache.org/activemq/browse/AMQ-1445
>>>>
>>>> FWIW, JDBC persistence works fine. Any ideas what might be wrong?
>>>>
>>>> Cheers,
>>>> David
>>>>
>>>> David Sitsky wrote:
>>>>> Hi Rob,
>>>>> Looks like the AMQ store works correctly - thanks for the advice.
>>>>> Cheers,
>>>>> David
>>>>> Rob Davies wrote:
>>>>>> Do you get the same using AMQ store (the default in 5.0) - its the
>>>>>> prefered store over kaha in ActiveMQ 5.0
>>>>>>
>>>>>>
>>>>>>
>>>>>> cheers,
>>>>>>
>>>>>> Rob
>>>>>>
>>>>>> http://rajdavies.blogspot.com/
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Oct 3, 2007, at 8:50 AM, David Sitsky wrote:
>>>>>>
>>>>>>> I have discovered an issue with the latest activemq 5.0 (revision
>>>>>>> 581510) which seems to have been present for some time, at least
>>>>>>> for the last month.
>>>>>>>
>>>>>>> I have an application using transactions, and kaha persistence,
>>>>>>> and two consumers feeding off a queue.
>>>>>>>
>>>>>>> I sometimes see the same message being sent to two consumers.
>>>>>>> All transactions were committed (no rollbacks), and as you can
>>>>>>> see by the debug dump, the message has the same message-ID, but
>>>>>>> was delivered to two separate consumers, running on separate
>>>>>>> threads, using sync receives with a prefetch of 0, with separate
>>>>>>> sessions of course.
>>>>>>>
>>>>>>> Is this a known problem? It doesn't always happen - seems to be
>>>>>>> a race condition. If I turn off persistence, I don't seem to see
>>>>>>> it.
>>>>>>>
>>>>>>> Here is the message being delivered to consumer 1:
>>>>>>>
>>>>>>> 2007-10-03 17:21:37,310 [Worker 060bacd87f4a40a4867113e2e3e553dd]
>>>>>>> 85475 DEBUG org.apache.activemq.ActiveMQMessageConsumer -
>>>>>>> ID:tamarama-62436-1191396093006-2:2:3:2 received message:
>>>>>>> MessageDispatch {commandId = 0, responseRequired = false,
>>>>>>> consumerId = ID:tamarama-62436-1191396093006-2:2:3:2, destination
>>>>>>> = queue://worker-items, message = ActiveMQObjectMessage
>>>>>>> {commandId = 50, responseRequired = false, messageId =
>>>>>>> ID:tamarama-62436-1191396093006-2:2:1:3:1, originalDestination =
>>>>>>> null, originalTransactionId = null, producerId =
>>>>>>> ID:tamarama-62436-1191396093006-2:2:1:3, destination =
>>>>>>> queue://worker-items, transactionId =
>>>>>>> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0,
>>>>>>> timestamp = 1191396097286, arrival = 0, brokerInTime =
>>>>>>> 1191396097290, brokerOutTime = 1191396097303, correlationId =
>>>>>>> null, replyTo = null, persistent = true, type = null, priority =
>>>>>>> 4, groupID = null, groupSequence = 0, targetConsumerId = null,
>>>>>>> compressed = false, userID = null, content =
>>>>>>> org.apache.activemq.util.ByteSequence@fa8cbc,
>>>>>>> marshalledProperties = null, dataStructure = null,
>>>>>>> redeliveryCounter = 0, size = 0, properties = null,
>>>>>>> readOnlyProperties = true, readOnlyBody = true, droppable =
>>>>>>> false}, redeliveryCounter = 0}
>>>>>>>
>>>>>>> Here is the same message (check the message ID) being delivered
>>>>>>> to consumer 2:
>>>>>>>
>>>>>>> 2007-10-03 17:21:37,422 [Worker 2c3169d1726b4781a423a239f65c7bd1]
>>>>>>> 85587 DEBUG org.apache.activemq.ActiveMQMessageConsumer -
>>>>>>> ID:tamarama-62436-1191396093006-2:2:1:2 received message:
>>>>>>> MessageDispatch {commandId = 0, responseRequired = false,
>>>>>>> consumerId = ID:tamarama-62436-1191396093006-2:2:1:2, destination
>>>>>>> = queue://worker-items, message = ActiveMQObjectMessage
>>>>>>> {commandId = 50, responseRequired = false, messageId =
>>>>>>> ID:tamarama-62436-1191396093006-2:2:1:3:1, originalDestination =
>>>>>>> null, originalTransactionId = null, producerId =
>>>>>>> ID:tamarama-62436-1191396093006-2:2:1:3, destination =
>>>>>>> queue://worker-items, transactionId =
>>>>>>> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0,
>>>>>>> timestamp = 1191396097286, arrival = 0, brokerInTime =
>>>>>>> 1191396097290, brokerOutTime = 1191396097417, correlationId =
>>>>>>> null, replyTo = null, persistent = true, type = null, priority =
>>>>>>> 4, groupID = null, groupSequence = 0, targetConsumerId = null,
>>>>>>> compressed = false, userID = null, content =
>>>>>>> org.apache.activemq.util.ByteSequence@1a881cc,
>>>>>>> marshalledProperties = null, dataStructure = null,
>>>>>>> redeliveryCounter = 0, size = 0, properties = null,
>>>>>>> readOnlyProperties = true, readOnlyBody = true, droppable =
>>>>>>> false}, redeliveryCounter = 0}
>>>>>>>
>>>>>>> I'll need to do more investigations tomorrow, but if anybody has
>>>>>>> any ideas, I'd appreciate some insights.
>>>>>>>
>>>>>>> --
>>>>>>> Cheers,
>>>>>>> David
>>>>>>>
>>>>>>> Nuix Pty Ltd
>>>>>>> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2
>>>>>>> 9280 0699
>>>>>>> Web: http://www.nuix.com Fax: +61 2
>>>>>>> 9212 6902
>>>>
>>>>
>>>> --
>>>> Cheers,
>>>> David
>>>>
>>>> Nuix Pty Ltd
>>>> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 9280
>>>> 0699
>>>> Web: http://www.nuix.com Fax: +61 2 9212
>>>> 6902
>>
>>
>> --
>> Cheers,
>> David
>>
>> Nuix Pty Ltd
>> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 9280 0699
>> Web: http://www.nuix.com Fax: +61 2 9212 6902
--
Cheers,
David
Nuix Pty Ltd
Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 9280 0699
Web: http://www.nuix.com Fax: +61 2 9212 6902
Re: RecoveryListenerAdapter - Message id ID... could not be recovered from the data store: WAS: [Duplicate message delivered to consumers on queue with kaha persistence]
Posted by Rob Davies <ra...@gmail.com>.
Hi David,
I had a test case that periodically could reproduce this - it now
can't after ensuring references are always freshed in the Kaha
reference store part of AMQ Store. Is your application now fixed?
thanks,
Rob
On Oct 8, 2007, at 2:48 AM, David Sitsky wrote:
> Hi Rob,
>
> Looks like you have fixed this issue - many thanks.
>
> Cheers,
> David
>
> Rob Davies wrote:
>> Hi David,
>> I've had a look through the log you've attached to AMQ-1445 - but
>> I think i'll need some more information to successfully resolve
>> whats going on. Obviously there's an attempt to retrieve a message
>> that's already been deleted - I just need to find out why ;)
>> Could I trouble you to try and devise a junit test case to reproduce?
>> thanks,
>> Rob
>> On Oct 5, 2007, at 7:37 AM, David Sitsky wrote:
>>> Hi Rob,
>>>
>>> I think I spoke too soon. While using AMQ store fixed the
>>> initial issue I was dealing with, it seems to have other issues
>>> in my application, where I see the following message before
>>> everything hangs:
>>>
>>> RecoveryListenerAdapter - Message id ID... could not be recovered
>>> from the data store!
>>>
>>> I created a JIRA record which contains the detailed debug message
>>> logs:
>>>
>>> https://issues.apache.org/activemq/browse/AMQ-1445
>>>
>>> FWIW, JDBC persistence works fine. Any ideas what might be wrong?
>>>
>>> Cheers,
>>> David
>>>
>>> David Sitsky wrote:
>>>> Hi Rob,
>>>> Looks like the AMQ store works correctly - thanks for the advice.
>>>> Cheers,
>>>> David
>>>> Rob Davies wrote:
>>>>> Do you get the same using AMQ store (the default in 5.0) - its
>>>>> the prefered store over kaha in ActiveMQ 5.0
>>>>>
>>>>>
>>>>>
>>>>> cheers,
>>>>>
>>>>> Rob
>>>>>
>>>>> http://rajdavies.blogspot.com/
>>>>>
>>>>>
>>>>>
>>>>> On Oct 3, 2007, at 8:50 AM, David Sitsky wrote:
>>>>>
>>>>>> I have discovered an issue with the latest activemq 5.0
>>>>>> (revision 581510) which seems to have been present for some
>>>>>> time, at least for the last month.
>>>>>>
>>>>>> I have an application using transactions, and kaha
>>>>>> persistence, and two consumers feeding off a queue.
>>>>>>
>>>>>> I sometimes see the same message being sent to two consumers.
>>>>>> All transactions were committed (no rollbacks), and as you can
>>>>>> see by the debug dump, the message has the same message-ID,
>>>>>> but was delivered to two separate consumers, running on
>>>>>> separate threads, using sync receives with a prefetch of 0,
>>>>>> with separate sessions of course.
>>>>>>
>>>>>> Is this a known problem? It doesn't always happen - seems to
>>>>>> be a race condition. If I turn off persistence, I don't seem
>>>>>> to see it.
>>>>>>
>>>>>> Here is the message being delivered to consumer 1:
>>>>>>
>>>>>> 2007-10-03 17:21:37,310 [Worker
>>>>>> 060bacd87f4a40a4867113e2e3e553dd] 85475 DEBUG
>>>>>> org.apache.activemq.ActiveMQMessageConsumer -
>>>>>> ID:tamarama-62436-1191396093006-2:2:3:2 received message:
>>>>>> MessageDispatch {commandId = 0, responseRequired = false,
>>>>>> consumerId = ID:tamarama-62436-1191396093006-2:2:3:2,
>>>>>> destination = queue://worker-items, message =
>>>>>> ActiveMQObjectMessage {commandId = 50, responseRequired =
>>>>>> false, messageId = ID:tamarama-62436-1191396093006-2:2:1:3:1,
>>>>>> originalDestination = null, originalTransactionId = null,
>>>>>> producerId = ID:tamarama-62436-1191396093006-2:2:1:3,
>>>>>> destination = queue://worker-items, transactionId =
>>>>>> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0,
>>>>>> timestamp = 1191396097286, arrival = 0, brokerInTime =
>>>>>> 1191396097290, brokerOutTime = 1191396097303, correlationId =
>>>>>> null, replyTo = null, persistent = true, type = null, priority
>>>>>> = 4, groupID = null, groupSequence = 0, targetConsumerId =
>>>>>> null, compressed = false, userID = null, content =
>>>>>> org.apache.activemq.util.ByteSequence@fa8cbc,
>>>>>> marshalledProperties = null, dataStructure = null,
>>>>>> redeliveryCounter = 0, size = 0, properties = null,
>>>>>> readOnlyProperties = true, readOnlyBody = true, droppable =
>>>>>> false}, redeliveryCounter = 0}
>>>>>>
>>>>>> Here is the same message (check the message ID) being
>>>>>> delivered to consumer 2:
>>>>>>
>>>>>> 2007-10-03 17:21:37,422 [Worker
>>>>>> 2c3169d1726b4781a423a239f65c7bd1] 85587 DEBUG
>>>>>> org.apache.activemq.ActiveMQMessageConsumer -
>>>>>> ID:tamarama-62436-1191396093006-2:2:1:2 received message:
>>>>>> MessageDispatch {commandId = 0, responseRequired = false,
>>>>>> consumerId = ID:tamarama-62436-1191396093006-2:2:1:2,
>>>>>> destination = queue://worker-items, message =
>>>>>> ActiveMQObjectMessage {commandId = 50, responseRequired =
>>>>>> false, messageId = ID:tamarama-62436-1191396093006-2:2:1:3:1,
>>>>>> originalDestination = null, originalTransactionId = null,
>>>>>> producerId = ID:tamarama-62436-1191396093006-2:2:1:3,
>>>>>> destination = queue://worker-items, transactionId =
>>>>>> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0,
>>>>>> timestamp = 1191396097286, arrival = 0, brokerInTime =
>>>>>> 1191396097290, brokerOutTime = 1191396097417, correlationId =
>>>>>> null, replyTo = null, persistent = true, type = null, priority
>>>>>> = 4, groupID = null, groupSequence = 0, targetConsumerId =
>>>>>> null, compressed = false, userID = null, content =
>>>>>> org.apache.activemq.util.ByteSequence@1a881cc,
>>>>>> marshalledProperties = null, dataStructure = null,
>>>>>> redeliveryCounter = 0, size = 0, properties = null,
>>>>>> readOnlyProperties = true, readOnlyBody = true, droppable =
>>>>>> false}, redeliveryCounter = 0}
>>>>>>
>>>>>> I'll need to do more investigations tomorrow, but if anybody
>>>>>> has any ideas, I'd appreciate some insights.
>>>>>>
>>>>>> --
>>>>>> Cheers,
>>>>>> David
>>>>>>
>>>>>> Nuix Pty Ltd
>>>>>> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2
>>>>>> 9280 0699
>>>>>> Web: http://www.nuix.com Fax: +61 2
>>>>>> 9212 6902
>>>
>>>
>>> --
>>> Cheers,
>>> David
>>>
>>> Nuix Pty Ltd
>>> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2
>>> 9280 0699
>>> Web: http://www.nuix.com Fax: +61 2
>>> 9212 6902
>
>
> --
> Cheers,
> David
>
> Nuix Pty Ltd
> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 9280
> 0699
> Web: http://www.nuix.com Fax: +61 2 9212
> 6902
Re: RecoveryListenerAdapter - Message id ID... could not be recovered
from the data store: WAS: [Duplicate message delivered to consumers on queue
with kaha persistence]
Posted by David Sitsky <si...@nuix.com>.
Hi Rob,
Looks like you have fixed this issue - many thanks.
Cheers,
David
Rob Davies wrote:
> Hi David,
>
> I've had a look through the log you've attached to AMQ-1445 - but I
> think i'll need some more information to successfully resolve whats
> going on. Obviously there's an attempt to retrieve a message that's
> already been deleted - I just need to find out why ;)
>
> Could I trouble you to try and devise a junit test case to reproduce?
>
> thanks,
>
> Rob
> On Oct 5, 2007, at 7:37 AM, David Sitsky wrote:
>
>> Hi Rob,
>>
>> I think I spoke too soon. While using AMQ store fixed the initial
>> issue I was dealing with, it seems to have other issues in my
>> application, where I see the following message before everything hangs:
>>
>> RecoveryListenerAdapter - Message id ID... could not be recovered from
>> the data store!
>>
>> I created a JIRA record which contains the detailed debug message logs:
>>
>> https://issues.apache.org/activemq/browse/AMQ-1445
>>
>> FWIW, JDBC persistence works fine. Any ideas what might be wrong?
>>
>> Cheers,
>> David
>>
>> David Sitsky wrote:
>>> Hi Rob,
>>> Looks like the AMQ store works correctly - thanks for the advice.
>>> Cheers,
>>> David
>>> Rob Davies wrote:
>>>> Do you get the same using AMQ store (the default in 5.0) - its the
>>>> prefered store over kaha in ActiveMQ 5.0
>>>>
>>>>
>>>>
>>>> cheers,
>>>>
>>>> Rob
>>>>
>>>> http://rajdavies.blogspot.com/
>>>>
>>>>
>>>>
>>>> On Oct 3, 2007, at 8:50 AM, David Sitsky wrote:
>>>>
>>>>> I have discovered an issue with the latest activemq 5.0 (revision
>>>>> 581510) which seems to have been present for some time, at least
>>>>> for the last month.
>>>>>
>>>>> I have an application using transactions, and kaha persistence, and
>>>>> two consumers feeding off a queue.
>>>>>
>>>>> I sometimes see the same message being sent to two consumers. All
>>>>> transactions were committed (no rollbacks), and as you can see by
>>>>> the debug dump, the message has the same message-ID, but was
>>>>> delivered to two separate consumers, running on separate threads,
>>>>> using sync receives with a prefetch of 0, with separate sessions of
>>>>> course.
>>>>>
>>>>> Is this a known problem? It doesn't always happen - seems to be a
>>>>> race condition. If I turn off persistence, I don't seem to see it.
>>>>>
>>>>> Here is the message being delivered to consumer 1:
>>>>>
>>>>> 2007-10-03 17:21:37,310 [Worker 060bacd87f4a40a4867113e2e3e553dd]
>>>>> 85475 DEBUG org.apache.activemq.ActiveMQMessageConsumer -
>>>>> ID:tamarama-62436-1191396093006-2:2:3:2 received message:
>>>>> MessageDispatch {commandId = 0, responseRequired = false,
>>>>> consumerId = ID:tamarama-62436-1191396093006-2:2:3:2, destination =
>>>>> queue://worker-items, message = ActiveMQObjectMessage {commandId =
>>>>> 50, responseRequired = false, messageId =
>>>>> ID:tamarama-62436-1191396093006-2:2:1:3:1, originalDestination =
>>>>> null, originalTransactionId = null, producerId =
>>>>> ID:tamarama-62436-1191396093006-2:2:1:3, destination =
>>>>> queue://worker-items, transactionId =
>>>>> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0, timestamp
>>>>> = 1191396097286, arrival = 0, brokerInTime = 1191396097290,
>>>>> brokerOutTime = 1191396097303, correlationId = null, replyTo =
>>>>> null, persistent = true, type = null, priority = 4, groupID = null,
>>>>> groupSequence = 0, targetConsumerId = null, compressed = false,
>>>>> userID = null, content =
>>>>> org.apache.activemq.util.ByteSequence@fa8cbc, marshalledProperties
>>>>> = null, dataStructure = null, redeliveryCounter = 0, size = 0,
>>>>> properties = null, readOnlyProperties = true, readOnlyBody = true,
>>>>> droppable = false}, redeliveryCounter = 0}
>>>>>
>>>>> Here is the same message (check the message ID) being delivered to
>>>>> consumer 2:
>>>>>
>>>>> 2007-10-03 17:21:37,422 [Worker 2c3169d1726b4781a423a239f65c7bd1]
>>>>> 85587 DEBUG org.apache.activemq.ActiveMQMessageConsumer -
>>>>> ID:tamarama-62436-1191396093006-2:2:1:2 received message:
>>>>> MessageDispatch {commandId = 0, responseRequired = false,
>>>>> consumerId = ID:tamarama-62436-1191396093006-2:2:1:2, destination =
>>>>> queue://worker-items, message = ActiveMQObjectMessage {commandId =
>>>>> 50, responseRequired = false, messageId =
>>>>> ID:tamarama-62436-1191396093006-2:2:1:3:1, originalDestination =
>>>>> null, originalTransactionId = null, producerId =
>>>>> ID:tamarama-62436-1191396093006-2:2:1:3, destination =
>>>>> queue://worker-items, transactionId =
>>>>> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0, timestamp
>>>>> = 1191396097286, arrival = 0, brokerInTime = 1191396097290,
>>>>> brokerOutTime = 1191396097417, correlationId = null, replyTo =
>>>>> null, persistent = true, type = null, priority = 4, groupID = null,
>>>>> groupSequence = 0, targetConsumerId = null, compressed = false,
>>>>> userID = null, content =
>>>>> org.apache.activemq.util.ByteSequence@1a881cc, marshalledProperties
>>>>> = null, dataStructure = null, redeliveryCounter = 0, size = 0,
>>>>> properties = null, readOnlyProperties = true, readOnlyBody = true,
>>>>> droppable = false}, redeliveryCounter = 0}
>>>>>
>>>>> I'll need to do more investigations tomorrow, but if anybody has
>>>>> any ideas, I'd appreciate some insights.
>>>>>
>>>>> --
>>>>> Cheers,
>>>>> David
>>>>>
>>>>> Nuix Pty Ltd
>>>>> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 9280
>>>>> 0699
>>>>> Web: http://www.nuix.com Fax: +61 2 9212
>>>>> 6902
>>
>>
>> --
>> Cheers,
>> David
>>
>> Nuix Pty Ltd
>> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 9280 0699
>> Web: http://www.nuix.com Fax: +61 2 9212 6902
--
Cheers,
David
Nuix Pty Ltd
Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 9280 0699
Web: http://www.nuix.com Fax: +61 2 9212 6902
Re: RecoveryListenerAdapter - Message id ID... could not be recovered from the data store: WAS: [Duplicate message delivered to consumers on queue with kaha persistence]
Posted by Rob Davies <ra...@gmail.com>.
Hi David,
I've had a look through the log you've attached to AMQ-1445 - but I
think i'll need some more information to successfully resolve whats
going on. Obviously there's an attempt to retrieve a message that's
already been deleted - I just need to find out why ;)
Could I trouble you to try and devise a junit test case to reproduce?
thanks,
Rob
On Oct 5, 2007, at 7:37 AM, David Sitsky wrote:
> Hi Rob,
>
> I think I spoke too soon. While using AMQ store fixed the initial
> issue I was dealing with, it seems to have other issues in my
> application, where I see the following message before everything
> hangs:
>
> RecoveryListenerAdapter - Message id ID... could not be recovered
> from the data store!
>
> I created a JIRA record which contains the detailed debug message
> logs:
>
> https://issues.apache.org/activemq/browse/AMQ-1445
>
> FWIW, JDBC persistence works fine. Any ideas what might be wrong?
>
> Cheers,
> David
>
> David Sitsky wrote:
>> Hi Rob,
>> Looks like the AMQ store works correctly - thanks for the advice.
>> Cheers,
>> David
>> Rob Davies wrote:
>>> Do you get the same using AMQ store (the default in 5.0) - its
>>> the prefered store over kaha in ActiveMQ 5.0
>>>
>>>
>>>
>>> cheers,
>>>
>>> Rob
>>>
>>> http://rajdavies.blogspot.com/
>>>
>>>
>>>
>>> On Oct 3, 2007, at 8:50 AM, David Sitsky wrote:
>>>
>>>> I have discovered an issue with the latest activemq 5.0
>>>> (revision 581510) which seems to have been present for some
>>>> time, at least for the last month.
>>>>
>>>> I have an application using transactions, and kaha persistence,
>>>> and two consumers feeding off a queue.
>>>>
>>>> I sometimes see the same message being sent to two consumers.
>>>> All transactions were committed (no rollbacks), and as you can
>>>> see by the debug dump, the message has the same message-ID, but
>>>> was delivered to two separate consumers, running on separate
>>>> threads, using sync receives with a prefetch of 0, with separate
>>>> sessions of course.
>>>>
>>>> Is this a known problem? It doesn't always happen - seems to be
>>>> a race condition. If I turn off persistence, I don't seem to
>>>> see it.
>>>>
>>>> Here is the message being delivered to consumer 1:
>>>>
>>>> 2007-10-03 17:21:37,310 [Worker
>>>> 060bacd87f4a40a4867113e2e3e553dd] 85475 DEBUG
>>>> org.apache.activemq.ActiveMQMessageConsumer -
>>>> ID:tamarama-62436-1191396093006-2:2:3:2 received message:
>>>> MessageDispatch {commandId = 0, responseRequired = false,
>>>> consumerId = ID:tamarama-62436-1191396093006-2:2:3:2,
>>>> destination = queue://worker-items, message =
>>>> ActiveMQObjectMessage {commandId = 50, responseRequired = false,
>>>> messageId = ID:tamarama-62436-1191396093006-2:2:1:3:1,
>>>> originalDestination = null, originalTransactionId = null,
>>>> producerId = ID:tamarama-62436-1191396093006-2:2:1:3,
>>>> destination = queue://worker-items, transactionId =
>>>> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0,
>>>> timestamp = 1191396097286, arrival = 0, brokerInTime =
>>>> 1191396097290, brokerOutTime = 1191396097303, correlationId =
>>>> null, replyTo = null, persistent = true, type = null, priority =
>>>> 4, groupID = null, groupSequence = 0, targetConsumerId = null,
>>>> compressed = false, userID = null, content =
>>>> org.apache.activemq.util.ByteSequence@fa8cbc,
>>>> marshalledProperties = null, dataStructure = null,
>>>> redeliveryCounter = 0, size = 0, properties = null,
>>>> readOnlyProperties = true, readOnlyBody = true, droppable =
>>>> false}, redeliveryCounter = 0}
>>>>
>>>> Here is the same message (check the message ID) being delivered
>>>> to consumer 2:
>>>>
>>>> 2007-10-03 17:21:37,422 [Worker
>>>> 2c3169d1726b4781a423a239f65c7bd1] 85587 DEBUG
>>>> org.apache.activemq.ActiveMQMessageConsumer -
>>>> ID:tamarama-62436-1191396093006-2:2:1:2 received message:
>>>> MessageDispatch {commandId = 0, responseRequired = false,
>>>> consumerId = ID:tamarama-62436-1191396093006-2:2:1:2,
>>>> destination = queue://worker-items, message =
>>>> ActiveMQObjectMessage {commandId = 50, responseRequired = false,
>>>> messageId = ID:tamarama-62436-1191396093006-2:2:1:3:1,
>>>> originalDestination = null, originalTransactionId = null,
>>>> producerId = ID:tamarama-62436-1191396093006-2:2:1:3,
>>>> destination = queue://worker-items, transactionId =
>>>> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0,
>>>> timestamp = 1191396097286, arrival = 0, brokerInTime =
>>>> 1191396097290, brokerOutTime = 1191396097417, correlationId =
>>>> null, replyTo = null, persistent = true, type = null, priority =
>>>> 4, groupID = null, groupSequence = 0, targetConsumerId = null,
>>>> compressed = false, userID = null, content =
>>>> org.apache.activemq.util.ByteSequence@1a881cc,
>>>> marshalledProperties = null, dataStructure = null,
>>>> redeliveryCounter = 0, size = 0, properties = null,
>>>> readOnlyProperties = true, readOnlyBody = true, droppable =
>>>> false}, redeliveryCounter = 0}
>>>>
>>>> I'll need to do more investigations tomorrow, but if anybody has
>>>> any ideas, I'd appreciate some insights.
>>>>
>>>> --
>>>> Cheers,
>>>> David
>>>>
>>>> Nuix Pty Ltd
>>>> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2
>>>> 9280 0699
>>>> Web: http://www.nuix.com Fax: +61 2
>>>> 9212 6902
>
>
> --
> Cheers,
> David
>
> Nuix Pty Ltd
> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 9280
> 0699
> Web: http://www.nuix.com Fax: +61 2 9212
> 6902
RecoveryListenerAdapter - Message id ID... could not be recovered
from the data store: WAS: [Duplicate message delivered to consumers on queue
with kaha persistence]
Posted by David Sitsky <si...@nuix.com>.
Hi Rob,
I think I spoke too soon. While using AMQ store fixed the initial issue
I was dealing with, it seems to have other issues in my application,
where I see the following message before everything hangs:
RecoveryListenerAdapter - Message id ID... could not be recovered from
the data store!
I created a JIRA record which contains the detailed debug message logs:
https://issues.apache.org/activemq/browse/AMQ-1445
FWIW, JDBC persistence works fine. Any ideas what might be wrong?
Cheers,
David
David Sitsky wrote:
> Hi Rob,
>
> Looks like the AMQ store works correctly - thanks for the advice.
>
> Cheers,
> David
>
> Rob Davies wrote:
>> Do you get the same using AMQ store (the default in 5.0) - its the
>> prefered store over kaha in ActiveMQ 5.0
>>
>>
>>
>> cheers,
>>
>> Rob
>>
>> http://rajdavies.blogspot.com/
>>
>>
>>
>> On Oct 3, 2007, at 8:50 AM, David Sitsky wrote:
>>
>>> I have discovered an issue with the latest activemq 5.0 (revision
>>> 581510) which seems to have been present for some time, at least for
>>> the last month.
>>>
>>> I have an application using transactions, and kaha persistence, and
>>> two consumers feeding off a queue.
>>>
>>> I sometimes see the same message being sent to two consumers. All
>>> transactions were committed (no rollbacks), and as you can see by the
>>> debug dump, the message has the same message-ID, but was delivered to
>>> two separate consumers, running on separate threads, using sync
>>> receives with a prefetch of 0, with separate sessions of course.
>>>
>>> Is this a known problem? It doesn't always happen - seems to be a
>>> race condition. If I turn off persistence, I don't seem to see it.
>>>
>>> Here is the message being delivered to consumer 1:
>>>
>>> 2007-10-03 17:21:37,310 [Worker 060bacd87f4a40a4867113e2e3e553dd]
>>> 85475 DEBUG org.apache.activemq.ActiveMQMessageConsumer -
>>> ID:tamarama-62436-1191396093006-2:2:3:2 received message:
>>> MessageDispatch {commandId = 0, responseRequired = false, consumerId
>>> = ID:tamarama-62436-1191396093006-2:2:3:2, destination =
>>> queue://worker-items, message = ActiveMQObjectMessage {commandId =
>>> 50, responseRequired = false, messageId =
>>> ID:tamarama-62436-1191396093006-2:2:1:3:1, originalDestination =
>>> null, originalTransactionId = null, producerId =
>>> ID:tamarama-62436-1191396093006-2:2:1:3, destination =
>>> queue://worker-items, transactionId =
>>> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0, timestamp =
>>> 1191396097286, arrival = 0, brokerInTime = 1191396097290,
>>> brokerOutTime = 1191396097303, correlationId = null, replyTo = null,
>>> persistent = true, type = null, priority = 4, groupID = null,
>>> groupSequence = 0, targetConsumerId = null, compressed = false,
>>> userID = null, content =
>>> org.apache.activemq.util.ByteSequence@fa8cbc, marshalledProperties =
>>> null, dataStructure = null, redeliveryCounter = 0, size = 0,
>>> properties = null, readOnlyProperties = true, readOnlyBody = true,
>>> droppable = false}, redeliveryCounter = 0}
>>>
>>> Here is the same message (check the message ID) being delivered to
>>> consumer 2:
>>>
>>> 2007-10-03 17:21:37,422 [Worker 2c3169d1726b4781a423a239f65c7bd1]
>>> 85587 DEBUG org.apache.activemq.ActiveMQMessageConsumer -
>>> ID:tamarama-62436-1191396093006-2:2:1:2 received message:
>>> MessageDispatch {commandId = 0, responseRequired = false, consumerId
>>> = ID:tamarama-62436-1191396093006-2:2:1:2, destination =
>>> queue://worker-items, message = ActiveMQObjectMessage {commandId =
>>> 50, responseRequired = false, messageId =
>>> ID:tamarama-62436-1191396093006-2:2:1:3:1, originalDestination =
>>> null, originalTransactionId = null, producerId =
>>> ID:tamarama-62436-1191396093006-2:2:1:3, destination =
>>> queue://worker-items, transactionId =
>>> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0, timestamp =
>>> 1191396097286, arrival = 0, brokerInTime = 1191396097290,
>>> brokerOutTime = 1191396097417, correlationId = null, replyTo = null,
>>> persistent = true, type = null, priority = 4, groupID = null,
>>> groupSequence = 0, targetConsumerId = null, compressed = false,
>>> userID = null, content =
>>> org.apache.activemq.util.ByteSequence@1a881cc, marshalledProperties =
>>> null, dataStructure = null, redeliveryCounter = 0, size = 0,
>>> properties = null, readOnlyProperties = true, readOnlyBody = true,
>>> droppable = false}, redeliveryCounter = 0}
>>>
>>> I'll need to do more investigations tomorrow, but if anybody has any
>>> ideas, I'd appreciate some insights.
>>>
>>> --
>>> Cheers,
>>> David
>>>
>>> Nuix Pty Ltd
>>> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 9280 0699
>>> Web: http://www.nuix.com Fax: +61 2 9212 6902
>
>
--
Cheers,
David
Nuix Pty Ltd
Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 9280 0699
Web: http://www.nuix.com Fax: +61 2 9212 6902
Re: Duplicate message delivered to consumers on queue with kaha
persistence
Posted by PieterN <se...@internethoofdkantoor.nl>.
PieterN wrote:
>
> This might be related to issue AMQ-1389, where I also see duplicate
> messages when using the Kaha persistence adapter, although in a bit
> different scenario. For me, the AMQ store works correctly as well, but I
> still believe this is a bug in the Kaha adapter.
>
Sorry I just see this issue is resolved -- haven't tried it yet though.
Pieter
--
View this message in context: http://www.nabble.com/Duplicate-message-delivered-to-consumers-on-queue-with-kaha-persistence-tf4560327s2354.html#a13034801
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Duplicate message delivered to consumers on queue with kaha
persistence
Posted by PieterN <se...@internethoofdkantoor.nl>.
David Sitsky-2 wrote:
>
> Looks like the AMQ store works correctly - thanks for the advice.
>
This might be related to issue AMQ-1389, where I also see duplicate messages
when using the Kaha persistence adapter, although in a bit different
scenario. For me, the AMQ store works correctly as well, but I still believe
this is a bug in the Kaha adapter.
Regards,
Pieter
--
View this message in context: http://www.nabble.com/Duplicate-message-delivered-to-consumers-on-queue-with-kaha-persistence-tf4560327s2354.html#a13034799
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Duplicate message delivered to consumers on queue with kaha persistence
Posted by David Sitsky <si...@nuix.com>.
Hi Rob,
Looks like the AMQ store works correctly - thanks for the advice.
Cheers,
David
Rob Davies wrote:
> Do you get the same using AMQ store (the default in 5.0) - its the
> prefered store over kaha in ActiveMQ 5.0
>
>
>
> cheers,
>
> Rob
>
> http://rajdavies.blogspot.com/
>
>
>
> On Oct 3, 2007, at 8:50 AM, David Sitsky wrote:
>
>> I have discovered an issue with the latest activemq 5.0 (revision
>> 581510) which seems to have been present for some time, at least for
>> the last month.
>>
>> I have an application using transactions, and kaha persistence, and
>> two consumers feeding off a queue.
>>
>> I sometimes see the same message being sent to two consumers. All
>> transactions were committed (no rollbacks), and as you can see by the
>> debug dump, the message has the same message-ID, but was delivered to
>> two separate consumers, running on separate threads, using sync
>> receives with a prefetch of 0, with separate sessions of course.
>>
>> Is this a known problem? It doesn't always happen - seems to be a
>> race condition. If I turn off persistence, I don't seem to see it.
>>
>> Here is the message being delivered to consumer 1:
>>
>> 2007-10-03 17:21:37,310 [Worker 060bacd87f4a40a4867113e2e3e553dd]
>> 85475 DEBUG org.apache.activemq.ActiveMQMessageConsumer -
>> ID:tamarama-62436-1191396093006-2:2:3:2 received message:
>> MessageDispatch {commandId = 0, responseRequired = false, consumerId =
>> ID:tamarama-62436-1191396093006-2:2:3:2, destination =
>> queue://worker-items, message = ActiveMQObjectMessage {commandId = 50,
>> responseRequired = false, messageId =
>> ID:tamarama-62436-1191396093006-2:2:1:3:1, originalDestination = null,
>> originalTransactionId = null, producerId =
>> ID:tamarama-62436-1191396093006-2:2:1:3, destination =
>> queue://worker-items, transactionId =
>> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0, timestamp =
>> 1191396097286, arrival = 0, brokerInTime = 1191396097290,
>> brokerOutTime = 1191396097303, correlationId = null, replyTo = null,
>> persistent = true, type = null, priority = 4, groupID = null,
>> groupSequence = 0, targetConsumerId = null, compressed = false, userID
>> = null, content = org.apache.activemq.util.ByteSequence@fa8cbc,
>> marshalledProperties = null, dataStructure = null, redeliveryCounter =
>> 0, size = 0, properties = null, readOnlyProperties = true,
>> readOnlyBody = true, droppable = false}, redeliveryCounter = 0}
>>
>> Here is the same message (check the message ID) being delivered to
>> consumer 2:
>>
>> 2007-10-03 17:21:37,422 [Worker 2c3169d1726b4781a423a239f65c7bd1]
>> 85587 DEBUG org.apache.activemq.ActiveMQMessageConsumer -
>> ID:tamarama-62436-1191396093006-2:2:1:2 received message:
>> MessageDispatch {commandId = 0, responseRequired = false, consumerId =
>> ID:tamarama-62436-1191396093006-2:2:1:2, destination =
>> queue://worker-items, message = ActiveMQObjectMessage {commandId = 50,
>> responseRequired = false, messageId =
>> ID:tamarama-62436-1191396093006-2:2:1:3:1, originalDestination = null,
>> originalTransactionId = null, producerId =
>> ID:tamarama-62436-1191396093006-2:2:1:3, destination =
>> queue://worker-items, transactionId =
>> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0, timestamp =
>> 1191396097286, arrival = 0, brokerInTime = 1191396097290,
>> brokerOutTime = 1191396097417, correlationId = null, replyTo = null,
>> persistent = true, type = null, priority = 4, groupID = null,
>> groupSequence = 0, targetConsumerId = null, compressed = false, userID
>> = null, content = org.apache.activemq.util.ByteSequence@1a881cc,
>> marshalledProperties = null, dataStructure = null, redeliveryCounter =
>> 0, size = 0, properties = null, readOnlyProperties = true,
>> readOnlyBody = true, droppable = false}, redeliveryCounter = 0}
>>
>> I'll need to do more investigations tomorrow, but if anybody has any
>> ideas, I'd appreciate some insights.
>>
>> --
>> Cheers,
>> David
>>
>> Nuix Pty Ltd
>> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 9280 0699
>> Web: http://www.nuix.com Fax: +61 2 9212 6902
--
Cheers,
David
Nuix Pty Ltd
Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 9280 0699
Web: http://www.nuix.com Fax: +61 2 9212 6902
Re: Duplicate message delivered to consumers on queue with kaha persistence
Posted by Rob Davies <ra...@gmail.com>.
Do you get the same using AMQ store (the default in 5.0) - its the
prefered store over kaha in ActiveMQ 5.0
cheers,
Rob
http://rajdavies.blogspot.com/
On Oct 3, 2007, at 8:50 AM, David Sitsky wrote:
> I have discovered an issue with the latest activemq 5.0 (revision
> 581510) which seems to have been present for some time, at least
> for the last month.
>
> I have an application using transactions, and kaha persistence, and
> two consumers feeding off a queue.
>
> I sometimes see the same message being sent to two consumers. All
> transactions were committed (no rollbacks), and as you can see by
> the debug dump, the message has the same message-ID, but was
> delivered to two separate consumers, running on separate threads,
> using sync receives with a prefetch of 0, with separate sessions of
> course.
>
> Is this a known problem? It doesn't always happen - seems to be a
> race condition. If I turn off persistence, I don't seem to see it.
>
> Here is the message being delivered to consumer 1:
>
> 2007-10-03 17:21:37,310 [Worker 060bacd87f4a40a4867113e2e3e553dd]
> 85475 DEBUG org.apache.activemq.ActiveMQMessageConsumer -
> ID:tamarama-62436-1191396093006-2:2:3:2 received message:
> MessageDispatch {commandId = 0, responseRequired = false,
> consumerId = ID:tamarama-62436-1191396093006-2:2:3:2, destination =
> queue://worker-items, message = ActiveMQObjectMessage {commandId =
> 50, responseRequired = false, messageId =
> ID:tamarama-62436-1191396093006-2:2:1:3:1, originalDestination =
> null, originalTransactionId = null, producerId =
> ID:tamarama-62436-1191396093006-2:2:1:3, destination = queue://
> worker-items, transactionId =
> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0, timestamp
> = 1191396097286, arrival = 0, brokerInTime = 1191396097290,
> brokerOutTime = 1191396097303, correlationId = null, replyTo =
> null, persistent = true, type = null, priority = 4, groupID = null,
> groupSequence = 0, targetConsumerId = null, compressed = false,
> userID = null, content =
> org.apache.activemq.util.ByteSequence@fa8cbc, marshalledProperties
> = null, dataStructure = null, redeliveryCounter = 0, size = 0,
> properties = null, readOnlyProperties = true, readOnlyBody = true,
> droppable = false}, redeliveryCounter = 0}
>
> Here is the same message (check the message ID) being delivered to
> consumer 2:
>
> 2007-10-03 17:21:37,422 [Worker 2c3169d1726b4781a423a239f65c7bd1]
> 85587 DEBUG org.apache.activemq.ActiveMQMessageConsumer -
> ID:tamarama-62436-1191396093006-2:2:1:2 received message:
> MessageDispatch {commandId = 0, responseRequired = false,
> consumerId = ID:tamarama-62436-1191396093006-2:2:1:2, destination =
> queue://worker-items, message = ActiveMQObjectMessage {commandId =
> 50, responseRequired = false, messageId =
> ID:tamarama-62436-1191396093006-2:2:1:3:1, originalDestination =
> null, originalTransactionId = null, producerId =
> ID:tamarama-62436-1191396093006-2:2:1:3, destination = queue://
> worker-items, transactionId =
> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0, timestamp
> = 1191396097286, arrival = 0, brokerInTime = 1191396097290,
> brokerOutTime = 1191396097417, correlationId = null, replyTo =
> null, persistent = true, type = null, priority = 4, groupID = null,
> groupSequence = 0, targetConsumerId = null, compressed = false,
> userID = null, content =
> org.apache.activemq.util.ByteSequence@1a881cc, marshalledProperties
> = null, dataStructure = null, redeliveryCounter = 0, size = 0,
> properties = null, readOnlyProperties = true, readOnlyBody = true,
> droppable = false}, redeliveryCounter = 0}
>
> I'll need to do more investigations tomorrow, but if anybody has
> any ideas, I'd appreciate some insights.
>
> --
> Cheers,
> David
>
> Nuix Pty Ltd
> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 9280
> 0699
> Web: http://www.nuix.com Fax: +61 2 9212
> 6902