You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@activemq.apache.org by "Lionel Cons (JIRA)" <ji...@apache.org> on 2012/06/25 12:49:42 UTC
[jira] [Created] (APLO-217) Broker does not always send all the
messages to concurrent topic consumers
Lionel Cons created APLO-217:
--------------------------------
Summary: Broker does not always send all the messages to concurrent topic consumers
Key: APLO-217
URL: https://issues.apache.org/jira/browse/APLO-217
Project: ActiveMQ Apollo
Issue Type: Bug
Environment: apollo-99-trunk-20120623.032010-57
Reporter: Lionel Cons
I am seeing problems with concurrent consumers on a topic which are quite similar to APLO-215. In this case however, adding a receipt header on DISCONNECT does not make the problem go away.
Here is the use case:
- Nc concurrent consumers on a topic (one per thread) that consume until they stop receiving new messages
- Np concurrent producers sending Nm messages to the topic (one per thread)
Each consumer should receive Np * Nm messages but, in practice, some seem to receive less. Extra care is taken to make sure things happen in order. For instance, producers start sending only after all consumers have received a RECEIPT back confirming the subscription. See the attached script for more information.
The results vary from one run to another. For instance, with Nc=Np=3 and Nm=10000:
[2] drained 30000 messages
[3] drained 30000 messages
[1] drained 30000 messages
(so this one is ok)
[2] drained 29983 messages
[3] drained 29983 messages
[1] drained 29998 messages
[1] drained 30000 messages
[3] drained 30000 messages
[2] drained 29980 messages
[3] drained 29998 messages
[1] drained 30000 messages
[2] drained 29976 messages
[2] drained 30000 messages
[1] drained 29980 messages
[3] drained 30000 messages
You may have to try with other values to reproduce the problem more reliably.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (APLO-217) Broker does not always send all the
messages to concurrent topic consumers
Posted by "Hiram Chirino (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/APLO-217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13400691#comment-13400691 ]
Hiram Chirino commented on APLO-217:
------------------------------------
If you need assured delivery messages, you should send each message with a receipt.
Having the disconnect wait for all in progress deliveries to complete/drain sounds like reasonable enhancement request. But in the mean time, just add receipt request for all your sends or at least the last send for each destination your sending to.
> Broker does not always send all the messages to concurrent topic consumers
> --------------------------------------------------------------------------
>
> Key: APLO-217
> URL: https://issues.apache.org/jira/browse/APLO-217
> Project: ActiveMQ Apollo
> Issue Type: Bug
> Environment: apollo-99-trunk-20120623.032010-57
> Reporter: Lionel Cons
> Attachments: APLO-217.pl
>
>
> I am seeing problems with concurrent consumers on a topic which are quite similar to APLO-215. In this case however, adding a receipt header on DISCONNECT does not make the problem go away.
> Here is the use case:
> - Nc concurrent consumers on a topic (one per thread) that consume until they stop receiving new messages
> - Np concurrent producers sending Nm messages to the topic (one per thread)
> Each consumer should receive Np * Nm messages but, in practice, some seem to receive less. Extra care is taken to make sure things happen in order. For instance, producers start sending only after all consumers have received a RECEIPT back confirming the subscription. See the attached script for more information.
> The results vary from one run to another. For instance, with Nc=Np=3 and Nm=10000:
> [2] drained 30000 messages
> [3] drained 30000 messages
> [1] drained 30000 messages
> (so this one is ok)
> [2] drained 29983 messages
> [3] drained 29983 messages
> [1] drained 29998 messages
> [1] drained 30000 messages
> [3] drained 30000 messages
> [2] drained 29980 messages
> [3] drained 29998 messages
> [1] drained 30000 messages
> [2] drained 29976 messages
> [2] drained 30000 messages
> [1] drained 29980 messages
> [3] drained 30000 messages
> You may have to try with other values to reproduce the problem more reliably.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (APLO-217) Broker does not always send all the
messages to concurrent topic consumers
Posted by "Hiram Chirino (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/APLO-217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401309#comment-13401309 ]
Hiram Chirino commented on APLO-217:
------------------------------------
Hi Lionel,
So waiting for the receipt on disconnect should be enough to assure the broker has received the message. I'm not really sure the disconnect was actually waiting for a receipt from the disconnect to come back before closing the socket. But that being said, it's important to note that waiting for the receipt on the disconnect is not enough to assure the the broker has persisted the message (in case of persistent messaging).
I'll add some some more test cases to verify this is actually the behavior.
> Broker does not always send all the messages to concurrent topic consumers
> --------------------------------------------------------------------------
>
> Key: APLO-217
> URL: https://issues.apache.org/jira/browse/APLO-217
> Project: ActiveMQ Apollo
> Issue Type: Bug
> Environment: apollo-99-trunk-20120623.032010-57
> Reporter: Lionel Cons
> Attachments: APLO-217.pl
>
>
> I am seeing problems with concurrent consumers on a topic which are quite similar to APLO-215. In this case however, adding a receipt header on DISCONNECT does not make the problem go away.
> Here is the use case:
> - Nc concurrent consumers on a topic (one per thread) that consume until they stop receiving new messages
> - Np concurrent producers sending Nm messages to the topic (one per thread)
> Each consumer should receive Np * Nm messages but, in practice, some seem to receive less. Extra care is taken to make sure things happen in order. For instance, producers start sending only after all consumers have received a RECEIPT back confirming the subscription. See the attached script for more information.
> The results vary from one run to another. For instance, with Nc=Np=3 and Nm=10000:
> [2] drained 30000 messages
> [3] drained 30000 messages
> [1] drained 30000 messages
> (so this one is ok)
> [2] drained 29983 messages
> [3] drained 29983 messages
> [1] drained 29998 messages
> [1] drained 30000 messages
> [3] drained 30000 messages
> [2] drained 29980 messages
> [3] drained 29998 messages
> [1] drained 30000 messages
> [2] drained 29976 messages
> [2] drained 30000 messages
> [1] drained 29980 messages
> [3] drained 30000 messages
> You may have to try with other values to reproduce the problem more reliably.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (APLO-217) Broker does not always send all the
messages to concurrent topic consumers
Posted by "Lionel Cons (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/APLO-217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13403013#comment-13403013 ]
Lionel Cons commented on APLO-217:
----------------------------------
After further debugging, APLO-217.pl does indeed show a problem... but not in Apollo.
The problem is on the client side with an incompatibility between select() to test data availability and SSL's internal buffering. I've added a workaround to Net::STOMP::Client (version >= 1.6_2, now in CPAN) and the test now passes with Apollo.
Thanks for your help debugging this.
> Broker does not always send all the messages to concurrent topic consumers
> --------------------------------------------------------------------------
>
> Key: APLO-217
> URL: https://issues.apache.org/jira/browse/APLO-217
> Project: ActiveMQ Apollo
> Issue Type: Bug
> Environment: apollo-99-trunk-20120623.032010-57
> Reporter: Lionel Cons
> Attachments: APLO-217.pl
>
>
> I am seeing problems with concurrent consumers on a topic which are quite similar to APLO-215. In this case however, adding a receipt header on DISCONNECT does not make the problem go away.
> Here is the use case:
> - Nc concurrent consumers on a topic (one per thread) that consume until they stop receiving new messages
> - Np concurrent producers sending Nm messages to the topic (one per thread)
> Each consumer should receive Np * Nm messages but, in practice, some seem to receive less. Extra care is taken to make sure things happen in order. For instance, producers start sending only after all consumers have received a RECEIPT back confirming the subscription. See the attached script for more information.
> The results vary from one run to another. For instance, with Nc=Np=3 and Nm=10000:
> [2] drained 30000 messages
> [3] drained 30000 messages
> [1] drained 30000 messages
> (so this one is ok)
> [2] drained 29983 messages
> [3] drained 29983 messages
> [1] drained 29998 messages
> [1] drained 30000 messages
> [3] drained 30000 messages
> [2] drained 29980 messages
> [3] drained 29998 messages
> [1] drained 30000 messages
> [2] drained 29976 messages
> [2] drained 30000 messages
> [1] drained 29980 messages
> [3] drained 30000 messages
> You may have to try with other values to reproduce the problem more reliably.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (APLO-217) Broker does not always send all the
messages to concurrent topic consumers
Posted by "Hiram Chirino (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/APLO-217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hiram Chirino resolved APLO-217.
--------------------------------
Resolution: Not A Problem
Assignee: Hiram Chirino
Thanks for the update. Closing as a non-issue.
> Broker does not always send all the messages to concurrent topic consumers
> --------------------------------------------------------------------------
>
> Key: APLO-217
> URL: https://issues.apache.org/jira/browse/APLO-217
> Project: ActiveMQ Apollo
> Issue Type: Bug
> Environment: apollo-99-trunk-20120623.032010-57
> Reporter: Lionel Cons
> Assignee: Hiram Chirino
> Attachments: APLO-217.pl
>
>
> I am seeing problems with concurrent consumers on a topic which are quite similar to APLO-215. In this case however, adding a receipt header on DISCONNECT does not make the problem go away.
> Here is the use case:
> - Nc concurrent consumers on a topic (one per thread) that consume until they stop receiving new messages
> - Np concurrent producers sending Nm messages to the topic (one per thread)
> Each consumer should receive Np * Nm messages but, in practice, some seem to receive less. Extra care is taken to make sure things happen in order. For instance, producers start sending only after all consumers have received a RECEIPT back confirming the subscription. See the attached script for more information.
> The results vary from one run to another. For instance, with Nc=Np=3 and Nm=10000:
> [2] drained 30000 messages
> [3] drained 30000 messages
> [1] drained 30000 messages
> (so this one is ok)
> [2] drained 29983 messages
> [3] drained 29983 messages
> [1] drained 29998 messages
> [1] drained 30000 messages
> [3] drained 30000 messages
> [2] drained 29980 messages
> [3] drained 29998 messages
> [1] drained 30000 messages
> [2] drained 29976 messages
> [2] drained 30000 messages
> [1] drained 29980 messages
> [3] drained 30000 messages
> You may have to try with other values to reproduce the problem more reliably.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (APLO-217) Broker does not always send all the
messages to concurrent topic consumers
Posted by "Lionel Cons (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/APLO-217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401385#comment-13401385 ]
Lionel Cons commented on APLO-217:
----------------------------------
FWIW, in Net::STOMP::Client (starting with version 1.6), the disconnect() method waits for the RECEIPT frame back if needed. So APLO-217.pl does show a problem.
> Broker does not always send all the messages to concurrent topic consumers
> --------------------------------------------------------------------------
>
> Key: APLO-217
> URL: https://issues.apache.org/jira/browse/APLO-217
> Project: ActiveMQ Apollo
> Issue Type: Bug
> Environment: apollo-99-trunk-20120623.032010-57
> Reporter: Lionel Cons
> Attachments: APLO-217.pl
>
>
> I am seeing problems with concurrent consumers on a topic which are quite similar to APLO-215. In this case however, adding a receipt header on DISCONNECT does not make the problem go away.
> Here is the use case:
> - Nc concurrent consumers on a topic (one per thread) that consume until they stop receiving new messages
> - Np concurrent producers sending Nm messages to the topic (one per thread)
> Each consumer should receive Np * Nm messages but, in practice, some seem to receive less. Extra care is taken to make sure things happen in order. For instance, producers start sending only after all consumers have received a RECEIPT back confirming the subscription. See the attached script for more information.
> The results vary from one run to another. For instance, with Nc=Np=3 and Nm=10000:
> [2] drained 30000 messages
> [3] drained 30000 messages
> [1] drained 30000 messages
> (so this one is ok)
> [2] drained 29983 messages
> [3] drained 29983 messages
> [1] drained 29998 messages
> [1] drained 30000 messages
> [3] drained 30000 messages
> [2] drained 29980 messages
> [3] drained 29998 messages
> [1] drained 30000 messages
> [2] drained 29976 messages
> [2] drained 30000 messages
> [1] drained 29980 messages
> [3] drained 30000 messages
> You may have to try with other values to reproduce the problem more reliably.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (APLO-217) Broker does not always send all the
messages to concurrent topic consumers
Posted by "Hiram Chirino (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/APLO-217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13400469#comment-13400469 ]
Hiram Chirino commented on APLO-217:
------------------------------------
Hi Lionel,
Could you add a receipt to the last message send, and then wait for that to complete before you shutdown the connection.
> Broker does not always send all the messages to concurrent topic consumers
> --------------------------------------------------------------------------
>
> Key: APLO-217
> URL: https://issues.apache.org/jira/browse/APLO-217
> Project: ActiveMQ Apollo
> Issue Type: Bug
> Environment: apollo-99-trunk-20120623.032010-57
> Reporter: Lionel Cons
> Attachments: APLO-217.pl
>
>
> I am seeing problems with concurrent consumers on a topic which are quite similar to APLO-215. In this case however, adding a receipt header on DISCONNECT does not make the problem go away.
> Here is the use case:
> - Nc concurrent consumers on a topic (one per thread) that consume until they stop receiving new messages
> - Np concurrent producers sending Nm messages to the topic (one per thread)
> Each consumer should receive Np * Nm messages but, in practice, some seem to receive less. Extra care is taken to make sure things happen in order. For instance, producers start sending only after all consumers have received a RECEIPT back confirming the subscription. See the attached script for more information.
> The results vary from one run to another. For instance, with Nc=Np=3 and Nm=10000:
> [2] drained 30000 messages
> [3] drained 30000 messages
> [1] drained 30000 messages
> (so this one is ok)
> [2] drained 29983 messages
> [3] drained 29983 messages
> [1] drained 29998 messages
> [1] drained 30000 messages
> [3] drained 30000 messages
> [2] drained 29980 messages
> [3] drained 29998 messages
> [1] drained 30000 messages
> [2] drained 29976 messages
> [2] drained 30000 messages
> [1] drained 29980 messages
> [3] drained 30000 messages
> You may have to try with other values to reproduce the problem more reliably.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (APLO-217) Broker does not always send all the
messages to concurrent topic consumers
Posted by "Lionel Cons (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/APLO-217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13400485#comment-13400485 ]
Lionel Cons commented on APLO-217:
----------------------------------
In this simplified use case I can add a receipt to the last message (and I will try this soon) but in real life use cases it is not always possible to know if a message being sent is the last one or not. Sometimes you have to look for a new one to send to realize that there are no more.
In any case, STOMP frames are received in order by the broker so isn't it enough to get a RECEIPT for frame n+1 (here = DISCONNECT) to be sure that frame n (here = last SEND) has been successfully processed by the broker? Imagine that the last SEND fails, shouldn't the broker send an ERROR frame for that failed SEND before the RECEIPT for the DISCONNECT?
> Broker does not always send all the messages to concurrent topic consumers
> --------------------------------------------------------------------------
>
> Key: APLO-217
> URL: https://issues.apache.org/jira/browse/APLO-217
> Project: ActiveMQ Apollo
> Issue Type: Bug
> Environment: apollo-99-trunk-20120623.032010-57
> Reporter: Lionel Cons
> Attachments: APLO-217.pl
>
>
> I am seeing problems with concurrent consumers on a topic which are quite similar to APLO-215. In this case however, adding a receipt header on DISCONNECT does not make the problem go away.
> Here is the use case:
> - Nc concurrent consumers on a topic (one per thread) that consume until they stop receiving new messages
> - Np concurrent producers sending Nm messages to the topic (one per thread)
> Each consumer should receive Np * Nm messages but, in practice, some seem to receive less. Extra care is taken to make sure things happen in order. For instance, producers start sending only after all consumers have received a RECEIPT back confirming the subscription. See the attached script for more information.
> The results vary from one run to another. For instance, with Nc=Np=3 and Nm=10000:
> [2] drained 30000 messages
> [3] drained 30000 messages
> [1] drained 30000 messages
> (so this one is ok)
> [2] drained 29983 messages
> [3] drained 29983 messages
> [1] drained 29998 messages
> [1] drained 30000 messages
> [3] drained 30000 messages
> [2] drained 29980 messages
> [3] drained 29998 messages
> [1] drained 30000 messages
> [2] drained 29976 messages
> [2] drained 30000 messages
> [1] drained 29980 messages
> [3] drained 30000 messages
> You may have to try with other values to reproduce the problem more reliably.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (APLO-217) Broker does not always send all the
messages to concurrent topic consumers
Posted by "Lionel Cons (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/APLO-217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401188#comment-13401188 ]
Lionel Cons commented on APLO-217:
----------------------------------
I find this confusing. In APLO-215 you suggest that adding a receipt to the DISCONNECT frame (and waiting for the RECEIPT frame back) is enough to make Apollo process the previously sent frames. For APLO-217, the same recipe does not seem to guarantee anymore the processing of the previously sent frames. The only difference is the type of frames sent before the DISCONNECT: ACK in APLO-215 versus SEND in APLO-217.
I think it's worth discussing this on the stomp-spec list. I'll start a thread on this matter.
> Broker does not always send all the messages to concurrent topic consumers
> --------------------------------------------------------------------------
>
> Key: APLO-217
> URL: https://issues.apache.org/jira/browse/APLO-217
> Project: ActiveMQ Apollo
> Issue Type: Bug
> Environment: apollo-99-trunk-20120623.032010-57
> Reporter: Lionel Cons
> Attachments: APLO-217.pl
>
>
> I am seeing problems with concurrent consumers on a topic which are quite similar to APLO-215. In this case however, adding a receipt header on DISCONNECT does not make the problem go away.
> Here is the use case:
> - Nc concurrent consumers on a topic (one per thread) that consume until they stop receiving new messages
> - Np concurrent producers sending Nm messages to the topic (one per thread)
> Each consumer should receive Np * Nm messages but, in practice, some seem to receive less. Extra care is taken to make sure things happen in order. For instance, producers start sending only after all consumers have received a RECEIPT back confirming the subscription. See the attached script for more information.
> The results vary from one run to another. For instance, with Nc=Np=3 and Nm=10000:
> [2] drained 30000 messages
> [3] drained 30000 messages
> [1] drained 30000 messages
> (so this one is ok)
> [2] drained 29983 messages
> [3] drained 29983 messages
> [1] drained 29998 messages
> [1] drained 30000 messages
> [3] drained 30000 messages
> [2] drained 29980 messages
> [3] drained 29998 messages
> [1] drained 30000 messages
> [2] drained 29976 messages
> [2] drained 30000 messages
> [1] drained 29980 messages
> [3] drained 30000 messages
> You may have to try with other values to reproduce the problem more reliably.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (APLO-217) Broker does not always send all the
messages to concurrent topic consumers
Posted by "Lionel Cons (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/APLO-217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lionel Cons updated APLO-217:
-----------------------------
Attachment: APLO-217.pl
> Broker does not always send all the messages to concurrent topic consumers
> --------------------------------------------------------------------------
>
> Key: APLO-217
> URL: https://issues.apache.org/jira/browse/APLO-217
> Project: ActiveMQ Apollo
> Issue Type: Bug
> Environment: apollo-99-trunk-20120623.032010-57
> Reporter: Lionel Cons
> Attachments: APLO-217.pl
>
>
> I am seeing problems with concurrent consumers on a topic which are quite similar to APLO-215. In this case however, adding a receipt header on DISCONNECT does not make the problem go away.
> Here is the use case:
> - Nc concurrent consumers on a topic (one per thread) that consume until they stop receiving new messages
> - Np concurrent producers sending Nm messages to the topic (one per thread)
> Each consumer should receive Np * Nm messages but, in practice, some seem to receive less. Extra care is taken to make sure things happen in order. For instance, producers start sending only after all consumers have received a RECEIPT back confirming the subscription. See the attached script for more information.
> The results vary from one run to another. For instance, with Nc=Np=3 and Nm=10000:
> [2] drained 30000 messages
> [3] drained 30000 messages
> [1] drained 30000 messages
> (so this one is ok)
> [2] drained 29983 messages
> [3] drained 29983 messages
> [1] drained 29998 messages
> [1] drained 30000 messages
> [3] drained 30000 messages
> [2] drained 29980 messages
> [3] drained 29998 messages
> [1] drained 30000 messages
> [2] drained 29976 messages
> [2] drained 30000 messages
> [1] drained 29980 messages
> [3] drained 30000 messages
> You may have to try with other values to reproduce the problem more reliably.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira