You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "Colin MacNaughton (JIRA)" <ji...@apache.org> on 2009/09/21 21:59:51 UTC
[jira] Created: (AMQ-2401) Hangs in fan-in to DUPS_OK_ACKNOWLEDGE
queue receivers
Hangs in fan-in to DUPS_OK_ACKNOWLEDGE queue receivers
------------------------------------------------------
Key: AMQ-2401
URL: https://issues.apache.org/activemq/browse/AMQ-2401
Project: ActiveMQ
Issue Type: Bug
Affects Versions: 5.3.0
Environment: N/A
Reporter: Colin MacNaughton
While running performance tests I I was seeing hangs in several tests involving dups ok queue receivers. My suspicion is that this is related to "too lazy" dups_ok acknowledgements. Changing the queue prefetchLimit to 100 caused this problem to go away. This needs more investigation, but it seems like we can get ourselves in to trouble if the queue size is smaller than the receiver's prefetchLimit, and this should be avoid. It is also possible that there is something more complicated happening in my tests. I haven't yet been able to reproduce this outside my performance test environment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-2401) Hangs in fan-in to DUPS_OK_ACKNOWLEDGE
queue receivers
Posted by "Colin MacNaughton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Colin MacNaughton updated AMQ-2401:
-----------------------------------
Environment:
Running with the following config:
{code:xml)
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="true" memoryLimit="1mb">
<pendingSubscriberPolicy>
<vmCursor/>
</pendingSubscriberPolicy>
</policyEntry>
<policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb"/>
</policyEntries>
</policyMap>
</destinationPolicy>
{code}
was:
Running with the following config:
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="true" memoryLimit="1mb">
<pendingSubscriberPolicy>
<vmCursor/>
</pendingSubscriberPolicy>
</policyEntry>
<policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb"/>
</policyEntries>
</policyMap>
</destinationPolicy>
> Hangs in fan-in to DUPS_OK_ACKNOWLEDGE queue receivers
> ------------------------------------------------------
>
> Key: AMQ-2401
> URL: https://issues.apache.org/activemq/browse/AMQ-2401
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.0
> Environment: Running with the following config:
> {code:xml)
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry topic=">" producerFlowControl="true" memoryLimit="1mb">
> <pendingSubscriberPolicy>
> <vmCursor/>
> </pendingSubscriberPolicy>
> </policyEntry>
> <policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb"/>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
> {code}
> Reporter: Colin MacNaughton
> Attachments: AMQ2401.txt
>
>
> While running performance tests I I was seeing hangs in several tests involving dups ok queue receivers. My suspicion is that this is related to "too lazy" dups_ok acknowledgements. Changing the queue prefetchLimit to 100 caused this problem to go away. This needs more investigation, but it seems like we can get ourselves in to trouble if the queue size is smaller than the receiver's prefetchLimit, and this should be avoid. It is also possible that there is something more complicated happening in my tests. I haven't yet been able to reproduce this outside my performance test environment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2401) Hangs in fan-in to DUPS_OK_ACKNOWLEDGE
queue receivers
Posted by "Colin MacNaughton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=54412#action_54412 ]
Colin MacNaughton commented on AMQ-2401:
----------------------------------------
The attached unit test can be used to demonstrate two issues. If you set PRODUCER_COUNT=1 and CONSUMER_COUNT=1, and change the producer message size to 1300 bytes. You can see that this will indeed produce a hang since the consumer's prefetch is 1000 and less than number fit into the queue.
However, using the test as is unmodified (e.g. 50 producers, and a message size of 1024 doesn't result in a hang, but does result in a serious performance bottleneck. There is some sort of contention happening in the broker. This is the behavior that I was seeing in my original performance runs ... simply a sever slow down.
> Hangs in fan-in to DUPS_OK_ACKNOWLEDGE queue receivers
> ------------------------------------------------------
>
> Key: AMQ-2401
> URL: https://issues.apache.org/activemq/browse/AMQ-2401
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.0
> Environment: Running with the following config:
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry topic=">" producerFlowControl="true" memoryLimit="1mb">
> <pendingSubscriberPolicy>
> <vmCursor/>
> </pendingSubscriberPolicy>
> </policyEntry>
> <policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb"/>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
> Reporter: Colin MacNaughton
> Attachments: AMQ2401.txt
>
>
> While running performance tests I I was seeing hangs in several tests involving dups ok queue receivers. My suspicion is that this is related to "too lazy" dups_ok acknowledgements. Changing the queue prefetchLimit to 100 caused this problem to go away. This needs more investigation, but it seems like we can get ourselves in to trouble if the queue size is smaller than the receiver's prefetchLimit, and this should be avoid. It is also possible that there is something more complicated happening in my tests. I haven't yet been able to reproduce this outside my performance test environment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2401) Hangs in fan-in to DUPS_OK_ACKNOWLEDGE
queue receivers
Posted by "Colin MacNaughton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=54435#action_54435 ]
Colin MacNaughton commented on AMQ-2401:
----------------------------------------
After some changes committed by Rob and Dejan in 5.4 trunk, the 50_1_1 usecase is better than before. However, setting the pending queue policy to vm results in the old buggy behavior, which is problematic for those thar don't wish to page messages to disk:
{code:title=Modify setup to include vm pending cursor,java}
entry.setPendingQueuePolicy(new VMPendingQueueMessageStoragePolicy());
{code}
> Hangs in fan-in to DUPS_OK_ACKNOWLEDGE queue receivers
> ------------------------------------------------------
>
> Key: AMQ-2401
> URL: https://issues.apache.org/activemq/browse/AMQ-2401
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.0
> Environment: Running with the following config:
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry topic=">" producerFlowControl="true" memoryLimit="1mb">
> <pendingSubscriberPolicy>
> <vmCursor/>
> </pendingSubscriberPolicy>
> </policyEntry>
> <policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb"/>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
> Reporter: Colin MacNaughton
> Attachments: AMQ2401.txt
>
>
> While running performance tests I I was seeing hangs in several tests involving dups ok queue receivers. My suspicion is that this is related to "too lazy" dups_ok acknowledgements. Changing the queue prefetchLimit to 100 caused this problem to go away. This needs more investigation, but it seems like we can get ourselves in to trouble if the queue size is smaller than the receiver's prefetchLimit, and this should be avoid. It is also possible that there is something more complicated happening in my tests. I haven't yet been able to reproduce this outside my performance test environment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2401) Hangs in fan-in to DUPS_OK_ACKNOWLEDGE
queue receivers
Posted by "Hiram Chirino (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=54471#action_54471 ]
Hiram Chirino commented on AMQ-2401:
------------------------------------
I think the safest bet will be to change the client so that he acks more frequently at least in a queue dups ok case.
> Hangs in fan-in to DUPS_OK_ACKNOWLEDGE queue receivers
> ------------------------------------------------------
>
> Key: AMQ-2401
> URL: https://issues.apache.org/activemq/browse/AMQ-2401
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.0
> Environment: Running with the following config:
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry topic=">" producerFlowControl="true" memoryLimit="1mb">
> <pendingSubscriberPolicy>
> <vmCursor/>
> </pendingSubscriberPolicy>
> </policyEntry>
> <policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb"/>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
> Reporter: Colin MacNaughton
> Attachments: AMQ2401.txt
>
>
> While running performance tests I I was seeing hangs in several tests involving dups ok queue receivers. My suspicion is that this is related to "too lazy" dups_ok acknowledgements. Changing the queue prefetchLimit to 100 caused this problem to go away. This needs more investigation, but it seems like we can get ourselves in to trouble if the queue size is smaller than the receiver's prefetchLimit, and this should be avoid. It is also possible that there is something more complicated happening in my tests. I haven't yet been able to reproduce this outside my performance test environment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-2401) Hangs in fan-in to DUPS_OK_ACKNOWLEDGE
queue receivers
Posted by "Hiram Chirino (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hiram Chirino updated AMQ-2401:
-------------------------------
Attachment: AMQ-2401.patch
Attaching patch which changes the DUPS_OK queue consumer behavior to match that of AUTO_ACK. The topic should still behave as before.
> Hangs in fan-in to DUPS_OK_ACKNOWLEDGE queue receivers
> ------------------------------------------------------
>
> Key: AMQ-2401
> URL: https://issues.apache.org/activemq/browse/AMQ-2401
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.0
> Environment: Running with the following config:
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry topic=">" producerFlowControl="true" memoryLimit="1mb">
> <pendingSubscriberPolicy>
> <vmCursor/>
> </pendingSubscriberPolicy>
> </policyEntry>
> <policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb"/>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
> Reporter: Colin MacNaughton
> Attachments: AMQ-2401.patch, AMQ2401.txt
>
>
> While running performance tests I I was seeing hangs in several tests involving dups ok queue receivers. My suspicion is that this is related to "too lazy" dups_ok acknowledgements. Changing the queue prefetchLimit to 100 caused this problem to go away. This needs more investigation, but it seems like we can get ourselves in to trouble if the queue size is smaller than the receiver's prefetchLimit, and this should be avoid. It is also possible that there is something more complicated happening in my tests. I haven't yet been able to reproduce this outside my performance test environment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (AMQ-2401) Hangs in fan-in to DUPS_OK_ACKNOWLEDGE
queue receivers
Posted by "Hiram Chirino (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hiram Chirino resolved AMQ-2401.
--------------------------------
Resolution: Fixed
Fix Version/s: 5.3.0
I added the test case to the build.. slightly modified so that it would exacerbate the DUPS_OK problem to get a quicker failure.
rev: 818496
The fix for the test case went in as rev: 818487
> Hangs in fan-in to DUPS_OK_ACKNOWLEDGE queue receivers
> ------------------------------------------------------
>
> Key: AMQ-2401
> URL: https://issues.apache.org/activemq/browse/AMQ-2401
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.0
> Environment: Running with the following config:
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry topic=">" producerFlowControl="true" memoryLimit="1mb">
> <pendingSubscriberPolicy>
> <vmCursor/>
> </pendingSubscriberPolicy>
> </policyEntry>
> <policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb"/>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
> Reporter: Colin MacNaughton
> Fix For: 5.3.0
>
> Attachments: AMQ-2401.patch, AMQ2401.txt
>
>
> While running performance tests I I was seeing hangs in several tests involving dups ok queue receivers. My suspicion is that this is related to "too lazy" dups_ok acknowledgements. Changing the queue prefetchLimit to 100 caused this problem to go away. This needs more investigation, but it seems like we can get ourselves in to trouble if the queue size is smaller than the receiver's prefetchLimit, and this should be avoid. It is also possible that there is something more complicated happening in my tests. I haven't yet been able to reproduce this outside my performance test environment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-2401) Hangs in fan-in to DUPS_OK_ACKNOWLEDGE
queue receivers
Posted by "Colin MacNaughton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Colin MacNaughton updated AMQ-2401:
-----------------------------------
Environment:
Running with the following config:
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="true" memoryLimit="1mb">
<pendingSubscriberPolicy>
<vmCursor/>
</pendingSubscriberPolicy>
</policyEntry>
<policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb"/>
</policyEntries>
</policyMap>
</destinationPolicy>
was:
Running with the following config:
{code:xml)
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="true" memoryLimit="1mb">
<pendingSubscriberPolicy>
<vmCursor/>
</pendingSubscriberPolicy>
</policyEntry>
<policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb"/>
</policyEntries>
</policyMap>
</destinationPolicy>
{code}
> Hangs in fan-in to DUPS_OK_ACKNOWLEDGE queue receivers
> ------------------------------------------------------
>
> Key: AMQ-2401
> URL: https://issues.apache.org/activemq/browse/AMQ-2401
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.0
> Environment: Running with the following config:
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry topic=">" producerFlowControl="true" memoryLimit="1mb">
> <pendingSubscriberPolicy>
> <vmCursor/>
> </pendingSubscriberPolicy>
> </policyEntry>
> <policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb"/>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
> Reporter: Colin MacNaughton
> Attachments: AMQ2401.txt
>
>
> While running performance tests I I was seeing hangs in several tests involving dups ok queue receivers. My suspicion is that this is related to "too lazy" dups_ok acknowledgements. Changing the queue prefetchLimit to 100 caused this problem to go away. This needs more investigation, but it seems like we can get ourselves in to trouble if the queue size is smaller than the receiver's prefetchLimit, and this should be avoid. It is also possible that there is something more complicated happening in my tests. I haven't yet been able to reproduce this outside my performance test environment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-2401) Hangs in fan-in to DUPS_OK_ACKNOWLEDGE
queue receivers
Posted by "Colin MacNaughton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Colin MacNaughton updated AMQ-2401:
-----------------------------------
Environment:
Running with the following config:
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="true" memoryLimit="1mb">
<pendingSubscriberPolicy>
<vmCursor/>
</pendingSubscriberPolicy>
</policyEntry>
<policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb"/>
</policyEntries>
</policyMap>
</destinationPolicy>
was:N/A
> Hangs in fan-in to DUPS_OK_ACKNOWLEDGE queue receivers
> ------------------------------------------------------
>
> Key: AMQ-2401
> URL: https://issues.apache.org/activemq/browse/AMQ-2401
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.0
> Environment: Running with the following config:
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry topic=">" producerFlowControl="true" memoryLimit="1mb">
> <pendingSubscriberPolicy>
> <vmCursor/>
> </pendingSubscriberPolicy>
> </policyEntry>
> <policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb"/>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
> Reporter: Colin MacNaughton
>
> While running performance tests I I was seeing hangs in several tests involving dups ok queue receivers. My suspicion is that this is related to "too lazy" dups_ok acknowledgements. Changing the queue prefetchLimit to 100 caused this problem to go away. This needs more investigation, but it seems like we can get ourselves in to trouble if the queue size is smaller than the receiver's prefetchLimit, and this should be avoid. It is also possible that there is something more complicated happening in my tests. I haven't yet been able to reproduce this outside my performance test environment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-2401) Hangs in fan-in to DUPS_OK_ACKNOWLEDGE
queue receivers
Posted by "Colin MacNaughton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Colin MacNaughton updated AMQ-2401:
-----------------------------------
Attachment: AMQ2401.txt
Path file for Unit Test Reproducing the issue.
> Hangs in fan-in to DUPS_OK_ACKNOWLEDGE queue receivers
> ------------------------------------------------------
>
> Key: AMQ-2401
> URL: https://issues.apache.org/activemq/browse/AMQ-2401
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.0
> Environment: Running with the following config:
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry topic=">" producerFlowControl="true" memoryLimit="1mb">
> <pendingSubscriberPolicy>
> <vmCursor/>
> </pendingSubscriberPolicy>
> </policyEntry>
> <policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb"/>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
> Reporter: Colin MacNaughton
> Attachments: AMQ2401.txt
>
>
> While running performance tests I I was seeing hangs in several tests involving dups ok queue receivers. My suspicion is that this is related to "too lazy" dups_ok acknowledgements. Changing the queue prefetchLimit to 100 caused this problem to go away. This needs more investigation, but it seems like we can get ourselves in to trouble if the queue size is smaller than the receiver's prefetchLimit, and this should be avoid. It is also possible that there is something more complicated happening in my tests. I haven't yet been able to reproduce this outside my performance test environment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.