You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by bonnyr <bo...@optusnet.com.au> on 2008/11/13 06:27:14 UTC
AMQ-5.2.0 RC3: JMS Exception Could not correlate acknowledgment
with dispatched message
All,
We've been using AMQ for a while now, and wanted to test the upcoming
release to see if it resolved
a problem we're having with the current version we're using (5.1.0), so I've
setup a test to try
and create the problem in my test environment.
My setup is:
-----------
1 producer, producing messages to a queue at a high rate (~1000 m/s). The
message payload is a
simple POJO we use for our project.
2 consumers connected to the same queue, one as primary and the other as
secondary (arbitrary
designation, but done such that the primary will be collecting the
messages if it's connected).
each consumer consumes messages in a batch (some glue layer in our app
does this), and simply
acknowledges the last message in the batch.
1 broker hosting the queue.
All 4 executables are running on the same host. All AMQ clients are written
in Java. Test is running on WinXP SP3
After executing for a little while, I am getting the exception in the title.
Once it occures, the simple
test clients are simply disconnected and hang, not able to receive any more.
Can someone suggest a way to try and pin this down? Is this a performance
problem? Is it related to
the fact that it's running on the same host?
Any help will be appreciated.
Cheers,
Bonny.
PS. the production/consumption rate seems to get stuck at around 1000 m/s
which is quite a bit lower
than when we originally tested (with 4.1.1) - any idea on what I can do to
improve this?
--
View this message in context: http://www.nabble.com/AMQ-5.2.0-RC3%3A-JMS-Exception-Could-not-correlate-acknowledgment-with-dispatched-message-tp20475017p20475017.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: AMQ-5.2.0 RC3: JMS Exception Could not correlate acknowledgment with dispatched message
Posted by Gary Tully <ga...@gmail.com>.
I think you are experiencing automatic producer flow control in
response to the memory utilisation limits, see:
http://activemq.apache.org/what-happens-with-a-fast-producer-and-slow-consumer.html
The details of how to disable are at:
http://activemq.apache.org/producer-flow-control.html
Just fyi, there is a bunch of good usage information hidden in:
http://activemq.apache.org/faq.html
On restart, the slowness of recovery, this is a known issue that is
being addresses for 6.0
There is a cleanup task that works with the check pointing that can
reduce the amount of work outstanding on a restart, they run every 30
seconds or so but they are only relevant when there is consumption of
messages. with a large backlog of unconsumed messages, the active data
file list grows and the required collection of indexes that need to be
rebuilt on restart (recovery) gets large.
There are a bunch of related configuration options (data and index
file size etc) on the Kaha store that may help some here, but the
underlying problem will not be addressed till 6.0.
2008/11/14 bonnyr <bo...@optusnet.com.au>:
>
> Gary,
>
> Thanks for the pointers. I'll try those. In the meantime I'm having other
> issues...
> I've cleared up the data directory in the broker area (we're using kaha for
> persistence)
> and restarted the test. I am not able to create the error again !?
> However, I've noticed another, more depressing problem :) - as long as there
> are consumers
> and producers, and the consumers keep up, I get a sustained rate that is
> reasonably stable. So far so
> good. When the consumer dies and the queue builds up, the rate gradually
> drops until (at around 1M msgs in the queue)
> it stops almost completly and will allow around 3 m/s. Connecting the
> consumer at this point does not
> help completely. In fact, the consumer is not able to consume at the maximum
> rate I've observed
> on the test rig (~1200 m/s) but rather at ~250 m/s and at that point the
> producer has recovered its
> rate to ~100 m/s. Why is that?
>
> Also, I've shut down the broker and restarted it (there were ~800K messages
> in the queue plus whatever
> other operations related to acks/deletions stored in the roll forward files)
> and it has taken so far 1h 20m
> to go through about 80% of the data files (in the journal directory) but
> what's more troubling is that
> the operation gets slower and slower, almost exponentially. Again, any idea
> as to why ?
>
> Cheers,
>
> Bonny
>
> Gary Tully wrote:
>>
>>> This test is the first time we're seeing this message - our applications
>>> are
>>> in production environments
>>> and this is not being exhibited at all, so I think our batch glue layer
>>> is
>>> not the source of the problem.
>>>
>> Ok, I am not thinking it is the source of the problem but it may be
>> the key to reproducing the problem in a test case.
>>
>>> Also, the code I'm using in the test uses other glue classes which are
>>> part
>>> of our framework and cannot
>>> be easily separated to form a stand alone test case. Could you perhaps
>>> point
>>> me to a test case that
>>> perhaps resembles my setup and I'll be happy to modify it to test what
>>> I'm
>>> trying to achieve?
>>>
>> Great. Possibly look at
>> http://svn.apache.org/repos/asf/activemq/trunk/activemq-core/src/test/java/org/apache/activemq/usecases/CompositePublishTest.java
>> The support class that it extends could provide some useful
>> scaffolding. Ie: this starts an embedded broker with consumers and
>> produces and validate that the produced messages were consumed.
>> Another example of the same sort of thing, creating consumers and
>> firing off loads of messages, asserting behavior is:
>> https://svn.apache.org/repos/asf/activemq/trunk/activemq-core/src/test/java/org/apache/activemq/JmsQueueSendReceiveUsingTwoSessionsTest.java
>>
>> Hope this helps get you started.
>>
>>
>
> --
> View this message in context: http://www.nabble.com/AMQ-5.2.0-RC3%3A-JMS-Exception-Could-not-correlate-acknowledgment-with-dispatched-message-tp20475017p20494692.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
Re: AMQ-5.2.0 RC3: JMS Exception Could not correlate acknowledgment
with dispatched message
Posted by bonnyr <bo...@optusnet.com.au>.
Gary,
Thanks for the pointers. I'll try those. In the meantime I'm having other
issues...
I've cleared up the data directory in the broker area (we're using kaha for
persistence)
and restarted the test. I am not able to create the error again !?
However, I've noticed another, more depressing problem :) - as long as there
are consumers
and producers, and the consumers keep up, I get a sustained rate that is
reasonably stable. So far so
good. When the consumer dies and the queue builds up, the rate gradually
drops until (at around 1M msgs in the queue)
it stops almost completly and will allow around 3 m/s. Connecting the
consumer at this point does not
help completely. In fact, the consumer is not able to consume at the maximum
rate I've observed
on the test rig (~1200 m/s) but rather at ~250 m/s and at that point the
producer has recovered its
rate to ~100 m/s. Why is that?
Also, I've shut down the broker and restarted it (there were ~800K messages
in the queue plus whatever
other operations related to acks/deletions stored in the roll forward files)
and it has taken so far 1h 20m
to go through about 80% of the data files (in the journal directory) but
what's more troubling is that
the operation gets slower and slower, almost exponentially. Again, any idea
as to why ?
Cheers,
Bonny
Gary Tully wrote:
>
>> This test is the first time we're seeing this message - our applications
>> are
>> in production environments
>> and this is not being exhibited at all, so I think our batch glue layer
>> is
>> not the source of the problem.
>>
> Ok, I am not thinking it is the source of the problem but it may be
> the key to reproducing the problem in a test case.
>
>> Also, the code I'm using in the test uses other glue classes which are
>> part
>> of our framework and cannot
>> be easily separated to form a stand alone test case. Could you perhaps
>> point
>> me to a test case that
>> perhaps resembles my setup and I'll be happy to modify it to test what
>> I'm
>> trying to achieve?
>>
> Great. Possibly look at
> http://svn.apache.org/repos/asf/activemq/trunk/activemq-core/src/test/java/org/apache/activemq/usecases/CompositePublishTest.java
> The support class that it extends could provide some useful
> scaffolding. Ie: this starts an embedded broker with consumers and
> produces and validate that the produced messages were consumed.
> Another example of the same sort of thing, creating consumers and
> firing off loads of messages, asserting behavior is:
> https://svn.apache.org/repos/asf/activemq/trunk/activemq-core/src/test/java/org/apache/activemq/JmsQueueSendReceiveUsingTwoSessionsTest.java
>
> Hope this helps get you started.
>
>
--
View this message in context: http://www.nabble.com/AMQ-5.2.0-RC3%3A-JMS-Exception-Could-not-correlate-acknowledgment-with-dispatched-message-tp20475017p20494692.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: AMQ-5.2.0 RC3: JMS Exception Could not correlate acknowledgment with dispatched message
Posted by Gary Tully <ga...@gmail.com>.
> This test is the first time we're seeing this message - our applications are
> in production environments
> and this is not being exhibited at all, so I think our batch glue layer is
> not the source of the problem.
>
Ok, I am not thinking it is the source of the problem but it may be
the key to reproducing the problem in a test case.
> Also, the code I'm using in the test uses other glue classes which are part
> of our framework and cannot
> be easily separated to form a stand alone test case. Could you perhaps point
> me to a test case that
> perhaps resembles my setup and I'll be happy to modify it to test what I'm
> trying to achieve?
>
Great. Possibly look at
http://svn.apache.org/repos/asf/activemq/trunk/activemq-core/src/test/java/org/apache/activemq/usecases/CompositePublishTest.java
The support class that it extends could provide some useful
scaffolding. Ie: this starts an embedded broker with consumers and
produces and validate that the produced messages were consumed.
Another example of the same sort of thing, creating consumers and
firing off loads of messages, asserting behavior is:
https://svn.apache.org/repos/asf/activemq/trunk/activemq-core/src/test/java/org/apache/activemq/JmsQueueSendReceiveUsingTwoSessionsTest.java
Hope this helps get you started.
Re: AMQ-5.2.0 RC3: JMS Exception Could not correlate acknowledgment
with dispatched message
Posted by bonnyr <bo...@optusnet.com.au>.
Gary,
This test is the first time we're seeing this message - our applications are
in production environments
and this is not being exhibited at all, so I think our batch glue layer is
not the source of the problem.
Also, the code I'm using in the test uses other glue classes which are part
of our framework and cannot
be easily separated to form a stand alone test case. Could you perhaps point
me to a test case that
perhaps resembles my setup and I'll be happy to modify it to test what I'm
trying to achieve?
Also, regarding the performance - I know we had obtained in similar setups a
much higher rate in the
past so I'm not sure what the issue there is, altough it currently is
secondary to this exception.
Cheers,
Bonny
Gary Tully wrote:
>
> I think the interesting bit is the batch logic in the consumers.
> Could you raise a jira and post your test case?
> If you could coerce the broker, producers and consumers into a single
> java application so that it can be converted into a junit test case it
> would be great.
>
> The key to pining this down is the reproducible test case. 1000 m/s
> and/or running on a single host should not be a problem.
>
> 2008/11/13 bonnyr <bo...@optusnet.com.au>:
>>
>> All,
>>
>> We've been using AMQ for a while now, and wanted to test the upcoming
>> release to see if it resolved
>> a problem we're having with the current version we're using (5.1.0), so
>> I've
>> setup a test to try
>> and create the problem in my test environment.
>>
>> My setup is:
>> -----------
>> 1 producer, producing messages to a queue at a high rate (~1000 m/s). The
>> message payload is a
>> simple POJO we use for our project.
>>
>> 2 consumers connected to the same queue, one as primary and the other as
>> secondary (arbitrary
>> designation, but done such that the primary will be collecting the
>> messages if it's connected).
>> each consumer consumes messages in a batch (some glue layer in our app
>> does this), and simply
>> acknowledges the last message in the batch.
>>
>> 1 broker hosting the queue.
>>
>> All 4 executables are running on the same host. All AMQ clients are
>> written
>> in Java. Test is running on WinXP SP3
>>
>>
>> After executing for a little while, I am getting the exception in the
>> title.
>> Once it occures, the simple
>> test clients are simply disconnected and hang, not able to receive any
>> more.
>>
>> Can someone suggest a way to try and pin this down? Is this a performance
>> problem? Is it related to
>> the fact that it's running on the same host?
>>
>> Any help will be appreciated.
>>
>> Cheers,
>>
>> Bonny.
>>
>> PS. the production/consumption rate seems to get stuck at around 1000 m/s
>> which is quite a bit lower
>> than when we originally tested (with 4.1.1) - any idea on what I can do
>> to
>> improve this?
>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/AMQ-5.2.0-RC3%3A-JMS-Exception-Could-not-correlate-acknowledgment-with-dispatched-message-tp20475017p20475017.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>
>
--
View this message in context: http://www.nabble.com/AMQ-5.2.0-RC3%3A-JMS-Exception-Could-not-correlate-acknowledgment-with-dispatched-message-tp20475017p20480533.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: AMQ-5.2.0 RC3: JMS Exception Could not correlate acknowledgment with dispatched message
Posted by Gary Tully <ga...@gmail.com>.
I think the interesting bit is the batch logic in the consumers.
Could you raise a jira and post your test case?
If you could coerce the broker, producers and consumers into a single
java application so that it can be converted into a junit test case it
would be great.
The key to pining this down is the reproducible test case. 1000 m/s
and/or running on a single host should not be a problem.
2008/11/13 bonnyr <bo...@optusnet.com.au>:
>
> All,
>
> We've been using AMQ for a while now, and wanted to test the upcoming
> release to see if it resolved
> a problem we're having with the current version we're using (5.1.0), so I've
> setup a test to try
> and create the problem in my test environment.
>
> My setup is:
> -----------
> 1 producer, producing messages to a queue at a high rate (~1000 m/s). The
> message payload is a
> simple POJO we use for our project.
>
> 2 consumers connected to the same queue, one as primary and the other as
> secondary (arbitrary
> designation, but done such that the primary will be collecting the
> messages if it's connected).
> each consumer consumes messages in a batch (some glue layer in our app
> does this), and simply
> acknowledges the last message in the batch.
>
> 1 broker hosting the queue.
>
> All 4 executables are running on the same host. All AMQ clients are written
> in Java. Test is running on WinXP SP3
>
>
> After executing for a little while, I am getting the exception in the title.
> Once it occures, the simple
> test clients are simply disconnected and hang, not able to receive any more.
>
> Can someone suggest a way to try and pin this down? Is this a performance
> problem? Is it related to
> the fact that it's running on the same host?
>
> Any help will be appreciated.
>
> Cheers,
>
> Bonny.
>
> PS. the production/consumption rate seems to get stuck at around 1000 m/s
> which is quite a bit lower
> than when we originally tested (with 4.1.1) - any idea on what I can do to
> improve this?
>
>
> --
> View this message in context: http://www.nabble.com/AMQ-5.2.0-RC3%3A-JMS-Exception-Could-not-correlate-acknowledgment-with-dispatched-message-tp20475017p20475017.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>