You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "Vadim Pesochinskiy (JIRA)" <ji...@apache.org> on 2006/08/01 19:29:23 UTC
[jira] Created: (AMQ-855) Add support for prefetchSize = 0
Add support for prefetchSize = 0
--------------------------------
Key: AMQ-855
URL: https://issues.apache.org/activemq/browse/AMQ-855
Project: ActiveMQ
Issue Type: New Feature
Components: Broker
Affects Versions: 4.0.2, 4.0.1, 4.0
Environment: any
Reporter: Vadim Pesochinskiy
Priority: Critical
Fix For: 4.2
This feature would enable to support following test case:
2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] Commented: (AMQ-855) Add support for prefetchSize = 0
Posted by Vadim Pesochinsky <Va...@mscibarra.com>.
Hi!
Is there any chance someone can give me a rough estimate of when this issue
can be looked at. I would really appreciate if someone reviewed the patch
and told me if there are issues with it. Right now I have time to resolve
them. Thanks.
- Vadim.
--
View this message in context: http://www.nabble.com/-jira--Created%3A-%28AMQ-855%29-Add-support-for-prefetchSize-%3D-0-tf2035055.html#a5952876
Sent from the ActiveMQ - Dev forum at Nabble.com.
Re: [jira] Commented: (AMQ-855) Add support for prefetchSize = 0
Posted by Vadim Pesochinsky <Va...@mscibarra.com>.
Hello!
Is there anything I can do to move forward with this patch? Thanks.
--
View this message in context: http://www.nabble.com/-jira--Created%3A-%28AMQ-855%29-Add-support-for-prefetchSize-%3D-0-tf2035055.html#a5835695
Sent from the ActiveMQ - Dev forum at Nabble.com.
Re: [jira] Commented: (AMQ-855) Add support for prefetchSize = 0
Posted by Vadim Pesochinsky <Va...@mscibarra.com>.
I submitted a patch yesterday and now the issue is resolved. Is that the
feedback for my patch :-) ? Thanks.
--
View this message in context: http://www.nabble.com/-jira--Created%3A-%28AMQ-855%29-Add-support-for-prefetchSize-%3D-0-tf2035055.html#a5817788
Sent from the ActiveMQ - Dev forum at Nabble.com.
[jira] Reopened: (AMQ-855) Add support for prefetchSize = 0
Posted by "Ivo Danihelka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=all ]
Ivo Danihelka reopened AMQ-855:
-------------------------------
There remains a bug to be fixed. The method MessageConsumer.receive() does not block at all when prefetchSize=0 is used.
The current code for ActiveMQMessageConsumer.receive() behaves like ActiveMQMessageConsumer.receiveNoWait(). The receive() could be fixed by using sendPullCommand(0) instead of sendPullCommand(-1). The sendPullCommand(0) works for me.
Thanks. My usage of activemq includes distributed slow consumers, so I'm very happy to be able to use zero prefetch.
Out of topic note:
It seems to me that it would be safer to use AtomicBoolean for field PrefetchSubscription.dispatching.
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0.2, 4.0.1, 4.0
> Environment: any
> Reporter: Vadim Pesochinskiy
> Assigned To: Hiram Chirino
> Priority: Critical
> Fix For: 4.1
>
> Attachments: ActiveMQMessageConsumer.java.patch3, ActiveMQMessageConsumer.patch, ActiveMQMessageConsumer.patch2, PrefetchSubscription.java.patch2, PrefetchSubscription.java.patch3, PrefetchSubscription.patch, region.QueueSubscription.java.patch, ZeroPrefetchConsumerTest.java.patch, ZeroPrefetchConsumerTest.java.patch3
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AMQ-855) Add support for prefetchSize = 0
Posted by "Hiram Chirino (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=comments#action_36893 ]
Hiram Chirino commented on AMQ-855:
-----------------------------------
we need to also build some test cases for this. Any takers?
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0, 4.0.1, 4.0.2
> Environment: any
> Reporter: Vadim Pesochinskiy
> Assigned To: Hiram Chirino
> Priority: Critical
> Fix For: 4.1
>
> Attachments: ActiveMQMessageConsumer.patch, ActiveMQMessageConsumer.patch2, PrefetchSubscription.patch
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (AMQ-855) Add support for prefetchSize = 0
Posted by "Vadim Pesochinskiy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=all ]
Vadim Pesochinskiy updated AMQ-855:
-----------------------------------
Attachment: ZeroPrefetchConsumerTest.java.patch3
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0, 4.0.1, 4.0.2
> Environment: any
> Reporter: Vadim Pesochinskiy
> Assigned To: Hiram Chirino
> Priority: Critical
> Fix For: 4.1
>
> Attachments: ActiveMQMessageConsumer.java.patch3, ActiveMQMessageConsumer.patch, ActiveMQMessageConsumer.patch2, PrefetchSubscription.java.patch2, PrefetchSubscription.java.patch3, PrefetchSubscription.patch, region.QueueSubscription.java.patch, ZeroPrefetchConsumerTest.java.patch, ZeroPrefetchConsumerTest.java.patch3
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Reopened: (AMQ-855) Add support for prefetchSize = 0
Posted by "Maxim Fateev (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=all ]
Maxim Fateev reopened AMQ-855:
------------------------------
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0, 4.0.1, 4.0.2
> Environment: any
> Reporter: Vadim Pesochinskiy
> Priority: Critical
> Fix For: 4.2
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AMQ-855) Add support for prefetchSize = 0
Posted by "Vadim Pesochinskiy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=comments#action_36926 ]
Vadim Pesochinskiy commented on AMQ-855:
----------------------------------------
Oh! there are too many attachments now. Latest addition is:
PrefetchSubscription.java.patch3
ActiveMQMessageConsumer.java.patch3
ZeroPrefetchConsumerTest.java.patch3
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0, 4.0.1, 4.0.2
> Environment: any
> Reporter: Vadim Pesochinskiy
> Assigned To: Hiram Chirino
> Priority: Critical
> Fix For: 4.1
>
> Attachments: ActiveMQMessageConsumer.java.patch3, ActiveMQMessageConsumer.patch, ActiveMQMessageConsumer.patch2, PrefetchSubscription.java.patch2, PrefetchSubscription.java.patch3, PrefetchSubscription.patch, region.QueueSubscription.java.patch, ZeroPrefetchConsumerTest.java.patch, ZeroPrefetchConsumerTest.java.patch3
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (AMQ-855) Add support for prefetchSize = 0
Posted by "Hiram Chirino (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=all ]
Hiram Chirino resolved AMQ-855.
-------------------------------
Resolution: Fixed
no further feedback.. assuming issue is now resolved.
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0, 4.0.1, 4.0.2
> Environment: any
> Reporter: Vadim Pesochinskiy
> Assigned To: Hiram Chirino
> Priority: Critical
> Fix For: 4.1
>
> Attachments: ActiveMQMessageConsumer.patch, ActiveMQMessageConsumer.patch2, PrefetchSubscription.patch, region.QueueSubscription.java.patch, ZeroPrefetchConsumerTest.java.patch
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (AMQ-855) Add support for prefetchSize = 0
Posted by "james strachan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=all ]
james strachan resolved AMQ-855.
--------------------------------
Resolution: Won't Fix
I still don't see a valid use case for this change. Instead I think these issues are more applicable to the problem you are trying to solve...
http://issues.apache.org/activemq/browse/AMQ-122
http://issues.apache.org/activemq/browse/AMQ-850
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0, 4.0.1, 4.0.2
> Environment: any
> Reporter: Vadim Pesochinskiy
> Priority: Critical
> Fix For: 4.2
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AMQ-855) Add support for prefetchSize = 0
Posted by "Vadim Pesochinskiy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=comments#action_36712 ]
Vadim Pesochinskiy commented on AMQ-855:
----------------------------------------
Well... I think there is a valid need ;-). I think that requesting not to create idle consumers is not valid.
The solution that you proposed does not work for me. In my case JMS does not satisfy the complicated on-the-fly changing priorities that my system has. This is why I have to fiddle with the queues and multiple consumers, some of which maybe idle in certain cases. I do not see why this is not a valid requirement. I am willing to contribute time to solve this issue.
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0, 4.0.1, 4.0.2
> Environment: any
> Reporter: Vadim Pesochinskiy
> Priority: Critical
> Fix For: 4.2
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (AMQ-855) Add support for prefetchSize = 0
Posted by "Hiram Chirino (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=all ]
Hiram Chirino resolved AMQ-855.
-------------------------------
Resolution: Fixed
Fix for 'sendPullCommand(0)' issue applied to trunk revision 464643
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0, 4.0.1, 4.0.2
> Environment: any
> Reporter: Vadim Pesochinskiy
> Assigned To: Hiram Chirino
> Priority: Critical
> Fix For: 4.1
>
> Attachments: ActiveMQMessageConsumer.java.patch3, ActiveMQMessageConsumer.patch, ActiveMQMessageConsumer.patch2, PrefetchSubscription.java.patch2, PrefetchSubscription.java.patch3, PrefetchSubscription.patch, region.QueueSubscription.java.patch, ZeroPrefetchConsumerTest.java.patch, ZeroPrefetchConsumerTest.java.patch3
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (AMQ-855) Add support for prefetchSize = 0
Posted by "Vadim Pesochinskiy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=all ]
Vadim Pesochinskiy updated AMQ-855:
-----------------------------------
Attachment: PrefetchSubscription.patch
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0.2, 4.0.1, 4.0
> Environment: any
> Reporter: Vadim Pesochinskiy
> Priority: Critical
> Fix For: 4.1
>
> Attachments: ActiveMQMessageConsumer.patch, PrefetchSubscription.patch
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AMQ-855) Add support for prefetchSize = 0
Posted by "Vadim Pesochinskiy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=comments#action_36782 ]
Vadim Pesochinskiy commented on AMQ-855:
----------------------------------------
James,
I added implementation of pulling consumer whereby pullMessage() returns a response. Could you please browse through it and give me a feedback? Patches are for file versions PrefetechSubscription.java 1.15 and 1.22 ActiveMQMessageConsumer.java
Current implementation does not work for me, because when recieve(long) call times out the prefetch window is increasing and we are back to where we started. Thanks a lot.
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0.2, 4.0.1, 4.0
> Environment: any
> Reporter: Vadim Pesochinskiy
> Priority: Critical
> Fix For: 4.1
>
> Attachments: ActiveMQMessageConsumer.patch, PrefetchSubscription.patch
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AMQ-855) Add support for prefetchSize = 0
Posted by "Vadim Pesochinskiy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=comments#action_36925 ]
Vadim Pesochinskiy commented on AMQ-855:
----------------------------------------
Hi Hiram,
Had to take change #1 out because it does not work when receive*() called multiple times before commit(). Instead I changed consumer not to pull if it has a message already in prefetch (which would be the case with rollback). Looks slick. Updated unit tests. Thanks.
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0, 4.0.1, 4.0.2
> Environment: any
> Reporter: Vadim Pesochinskiy
> Assigned To: Hiram Chirino
> Priority: Critical
> Fix For: 4.1
>
> Attachments: ActiveMQMessageConsumer.java.patch3, ActiveMQMessageConsumer.patch, ActiveMQMessageConsumer.patch2, PrefetchSubscription.java.patch2, PrefetchSubscription.java.patch3, PrefetchSubscription.patch, region.QueueSubscription.java.patch, ZeroPrefetchConsumerTest.java.patch, ZeroPrefetchConsumerTest.java.patch3
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AMQ-855) Add support for prefetchSize = 0
Posted by "james strachan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=comments#action_36716 ]
james strachan commented on AMQ-855:
------------------------------------
Andrew I just commented on this thread on why a prefetch of 1 is the lowest possible value to have while still fully implementing the JMS spec...
http://www.nabble.com/forum/ViewPost.jtp?post=5583273&framed=y
A few further comments...
> Now, you could in theory hack together a "pull" model by setting prefetch size = 0, so that basically each ack is a request for the next message
Thats a prefetch of 1. With a prefetch of zero no messages would be dispatched so there could be no ack :)
> Systems like this need to be designed under the assumption that clients will not behave themselves.
> They will deadlock themselves, slow down arbitrarily, boxes will go up and down, etc.
> These things happen all the time in real life and shouldn't have adverse effects on other, well-behaved consumers.
This is a valid problem - a badly behaving consumer can hog a message. However changing from a push to pull model or having prefetch of 0 or 1 will not change this. A hogged message is a hogged message however the consumer manages to get the message (pull or push).
e.g. if we did implement prefetch of zero - which means don't deliver a message to a consumer at all - unless they perform a consumer.receive() - even then, the consumer could then hang/deadlock and never actually acknowledge or process the message.
The workaround to this is to just kill consumers if they take too long to process a message - see this JIRA which I think what you really need
http://issues.apache.org/activemq/browse/AMQ-850
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0, 4.0.1, 4.0.2
> Environment: any
> Reporter: Vadim Pesochinskiy
> Priority: Critical
> Fix For: 4.2
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (AMQ-855) Add support for prefetchSize = 0
Posted by "Hiram Chirino (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=all ]
Hiram Chirino resolved AMQ-855.
-------------------------------
Resolution: Fixed
patches applied.. Thanks Vadim!
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0, 4.0.1, 4.0.2
> Environment: any
> Reporter: Vadim Pesochinskiy
> Assigned To: Hiram Chirino
> Priority: Critical
> Fix For: 4.1
>
> Attachments: ActiveMQMessageConsumer.java.patch3, ActiveMQMessageConsumer.patch, ActiveMQMessageConsumer.patch2, PrefetchSubscription.java.patch2, PrefetchSubscription.java.patch3, PrefetchSubscription.patch, region.QueueSubscription.java.patch, ZeroPrefetchConsumerTest.java.patch, ZeroPrefetchConsumerTest.java.patch3
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AMQ-855) Add support for prefetchSize = 0
Posted by "Vadim Pesochinskiy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=comments#action_36738 ]
Vadim Pesochinskiy commented on AMQ-855:
----------------------------------------
#1. I thought that pending holds references, not messages. When message is actually dispatched it will become unavailable to other PrefetchSubscriptions. Maybe I am wrong.
#2. I thought that push was employed primarely to boost performance.
Whatever the reasons are, if you feel that this should be resolve could you please vote for it? Thanks a lot. Vadim.
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0, 4.0.1, 4.0.2
> Environment: any
> Reporter: Vadim Pesochinskiy
> Priority: Critical
> Fix For: 4.2
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AMQ-855) Add support for prefetchSize = 0
Posted by "Vadim Pesochinskiy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=comments#action_36895 ]
Vadim Pesochinskiy commented on AMQ-855:
----------------------------------------
Looking good Hiram. Attached is updated unit test so far and a fix for RuntimeException "Not implemented" when calling getGroupID() on the NULL_MESSAGE object.
For some reason when I change to transacted session none of the tests work (including test written by James and if prefetch is > 0). I did not get to the bottom of this yet.
Thanks a lot, great job.
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0, 4.0.1, 4.0.2
> Environment: any
> Reporter: Vadim Pesochinskiy
> Assigned To: Hiram Chirino
> Priority: Critical
> Fix For: 4.1
>
> Attachments: ActiveMQMessageConsumer.patch, ActiveMQMessageConsumer.patch2, PrefetchSubscription.patch, region.QueueSubscription.java.patch, ZeroPrefetchConsumerTest.java.patch
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (AMQ-855) Add support for prefetchSize = 0
Posted by "Vadim Pesochinskiy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=all ]
Vadim Pesochinskiy updated AMQ-855:
-----------------------------------
Attachment: region.QueueSubscription.java.patch
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0, 4.0.1, 4.0.2
> Environment: any
> Reporter: Vadim Pesochinskiy
> Assigned To: Hiram Chirino
> Priority: Critical
> Fix For: 4.1
>
> Attachments: ActiveMQMessageConsumer.patch, ActiveMQMessageConsumer.patch2, PrefetchSubscription.patch, region.QueueSubscription.java.patch
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AMQ-855) Add support for prefetchSize = 0
Posted by "Maxim Fateev (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=comments#action_36739 ]
Maxim Fateev commented on AMQ-855:
----------------------------------
#1. I wasn't presize. Yes it holds references. But it doesn't really matter as these references are locked to the particular subsription and are not reassigned while this subscription is alive.
#2. I think prefetch buffers are used to boost peformance. Pull model doesn't preclude their use. The question is if messages are assigned to subscriptions up front or only when space is available in prefetch buffers.
I've already voted for it.
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0, 4.0.1, 4.0.2
> Environment: any
> Reporter: Vadim Pesochinskiy
> Priority: Critical
> Fix For: 4.2
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (AMQ-855) Add support for prefetchSize = 0
Posted by "Vadim Pesochinskiy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=all ]
Vadim Pesochinskiy updated AMQ-855:
-----------------------------------
Attachment: ZeroPrefetchConsumerTest.java.patch
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0, 4.0.1, 4.0.2
> Environment: any
> Reporter: Vadim Pesochinskiy
> Assigned To: Hiram Chirino
> Priority: Critical
> Fix For: 4.1
>
> Attachments: ActiveMQMessageConsumer.patch, ActiveMQMessageConsumer.patch2, PrefetchSubscription.patch, region.QueueSubscription.java.patch, ZeroPrefetchConsumerTest.java.patch
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AMQ-855) Add support for prefetchSize = 0
Posted by "Vadim Pesochinskiy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=comments#action_36714 ]
Vadim Pesochinskiy commented on AMQ-855:
----------------------------------------
I agree this functionality is essential for slow consumers. I do not see the hacking part in it. If prefetchSize is set to 0 client cannot expect messages dispatched to it, so it has to send an ack to get next msg.
Of course push will work better when a lot of messages are sent. In this case AMQ-850 will fix it by timing out and disabling passive consumers.
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0, 4.0.1, 4.0.2
> Environment: any
> Reporter: Vadim Pesochinskiy
> Priority: Critical
> Fix For: 4.2
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (AMQ-855) Add support for prefetchSize = 0
Posted by "james strachan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=all ]
james strachan updated AMQ-855:
-------------------------------
Patch Info: [Patch Available]
setting the patch available flag
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0, 4.0.1, 4.0.2
> Environment: any
> Reporter: Vadim Pesochinskiy
> Priority: Critical
> Fix For: 4.1
>
> Attachments: ActiveMQMessageConsumer.patch, ActiveMQMessageConsumer.patch2, PrefetchSubscription.patch
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AMQ-855) Add support for prefetchSize = 0
Posted by "Hiram Chirino (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=comments#action_36711 ]
Hiram Chirino commented on AMQ-855:
-----------------------------------
I don't think this is a valid need.
If each server/worker thread only creates 1 consumer with a prefetch of 1 then the jobs will be proccessed in 11 minutes.
I think the problem is occuring becuse each sever/work thread is creating multiple consumers and each has a prefetch. Mutliple consumers can be avoided by using consumers that use destination wild cards.
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0, 4.0.1, 4.0.2
> Environment: any
> Reporter: Vadim Pesochinskiy
> Priority: Critical
> Fix For: 4.2
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (AMQ-855) Add support for prefetchSize = 0
Posted by "Vadim Pesochinskiy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=all ]
Vadim Pesochinskiy updated AMQ-855:
-----------------------------------
Attachment: ActiveMQMessageConsumer.java.patch3
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0, 4.0.1, 4.0.2
> Environment: any
> Reporter: Vadim Pesochinskiy
> Assigned To: Hiram Chirino
> Priority: Critical
> Fix For: 4.1
>
> Attachments: ActiveMQMessageConsumer.java.patch3, ActiveMQMessageConsumer.patch, ActiveMQMessageConsumer.patch2, PrefetchSubscription.java.patch2, PrefetchSubscription.patch, region.QueueSubscription.java.patch, ZeroPrefetchConsumerTest.java.patch
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AMQ-855) Add support for prefetchSize = 0
Posted by "Vadim Pesochinskiy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=comments#action_37198 ]
Vadim Pesochinskiy commented on AMQ-855:
----------------------------------------
I think you are right.
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0.2, 4.0.1, 4.0
> Environment: any
> Reporter: Vadim Pesochinskiy
> Assigned To: Hiram Chirino
> Priority: Critical
> Fix For: 4.1
>
> Attachments: ActiveMQMessageConsumer.java.patch3, ActiveMQMessageConsumer.patch, ActiveMQMessageConsumer.patch2, PrefetchSubscription.java.patch2, PrefetchSubscription.java.patch3, PrefetchSubscription.patch, region.QueueSubscription.java.patch, ZeroPrefetchConsumerTest.java.patch, ZeroPrefetchConsumerTest.java.patch3
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (AMQ-855) Add support for prefetchSize = 0
Posted by "james strachan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=all ]
james strachan resolved AMQ-855.
--------------------------------
Fix Version/s: 4.1
(was: 4.2)
Resolution: Fixed
I've just committed an implementation to the main trunk if you want to try it out - or take a look at the test case ZeroPrefetchConsumerTest.
So basically prefetch of zero can be supported for synchronous consumers (where receive(long) or receive() is called). We throw an exception if you try and use it for async subscription (consumer.setMessageListener()).
The one issue with the current implementation is that the receive(timeout) must wait long enough for a message to arrive from the broker; otherwise the method may return but a message is dispatched just shortly afterwards. So provided you give a reasonably large receive(timeout) then it should do what you need.
Note that the receiveNoWait() method does not try to pull messages from the broker since the method is not meant to wait for one to arrive.
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0, 4.0.1, 4.0.2
> Environment: any
> Reporter: Vadim Pesochinskiy
> Priority: Critical
> Fix For: 4.1
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (AMQ-855) Add support for prefetchSize = 0
Posted by "Vadim Pesochinskiy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=all ]
Vadim Pesochinskiy updated AMQ-855:
-----------------------------------
Attachment: ActiveMQMessageConsumer.patch2
Forgot to add handling for expired messages. Please discard previous patch and use this one instead.
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0, 4.0.1, 4.0.2
> Environment: any
> Reporter: Vadim Pesochinskiy
> Priority: Critical
> Fix For: 4.1
>
> Attachments: ActiveMQMessageConsumer.patch, ActiveMQMessageConsumer.patch2, PrefetchSubscription.patch
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AMQ-855) Add support for prefetchSize = 0
Posted by "Maxim Fateev (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=comments#action_36737 ]
Maxim Fateev commented on AMQ-855:
----------------------------------
I think the failure scenario (slow consumer) is very common one and should be supported by ActiveMQ. The root cause of the problem is not the prefetch size. As James pointed out setting it to one is good enough. But the problem is that messages are assigned to consumers as soon as they are dispatched. If consuemer's prefetch buffer is full then they are kept in "pending" list (see PrefetchSubscription) implementation. This list is essentially unlimited. So messages assigned to slow consumer are never redispatched to overs unless slow consumer is disconnected or the whole broker is bounced.
Standard way to solve this issue is to use "pull" model instead of "push". I.e. messages are not assigned to consumers up front but given to them when requested. I understand that "push" dispatch model of ActiveMQ is due to desire to support "selector expressions". But it doesn't mean that product should be unusable for customers that do not use selectors at all.
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0, 4.0.1, 4.0.2
> Environment: any
> Reporter: Vadim Pesochinskiy
> Priority: Critical
> Fix For: 4.2
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Reopened: (AMQ-855) Add support for prefetchSize = 0
Posted by "james strachan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=all ]
james strachan reopened AMQ-855:
--------------------------------
The issue was resolved before Vadim attached more patches
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0, 4.0.1, 4.0.2
> Environment: any
> Reporter: Vadim Pesochinskiy
> Priority: Critical
> Fix For: 4.1
>
> Attachments: ActiveMQMessageConsumer.patch, ActiveMQMessageConsumer.patch2, PrefetchSubscription.patch
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AMQ-855) Add support for prefetchSize = 0
Posted by "james strachan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=comments#action_36718 ]
james strachan commented on AMQ-855:
------------------------------------
BTW just to be absolutely clear - here's how I think we should fix Vadim's use case.
If we had full support for priority based dispatching so that...
* all high priority messages are dispatched before *ANY* medium priority.
* all medium priority messages are dispatched before *ANY* low priority
* all low priority messages are then dispatched
Then the consumers all have a prefetch of 1 and each consumer subscribes to high, medium and low priority messages.
The reason I don't see the prefetch of zero fixing the issue is as follows..
imagine the client did this...
message = highConsumer.receive(timeout);
if (message == null) {
// a high message could have just arrived now...
message = mediumConsumer.receive(timeout);
// a high message could have just arrived now...
}
if (message == null) {
// a high message could have just arrived now...
message = lowConsumer.receive(timeout);
// a high message could have just arrived now...
}
// a high message could have just arrived now...
there's still a chance the client would miss the high priority messages arriving as they will spend a reasonable amount of time in wait loops waiting to request-response to the broker.
The only way to truly guarrentee ordering is to get the broker to do it.
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0, 4.0.1, 4.0.2
> Environment: any
> Reporter: Vadim Pesochinskiy
> Priority: Critical
> Fix For: 4.2
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (AMQ-855) Add support for prefetchSize = 0
Posted by "Vadim Pesochinskiy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=all ]
Vadim Pesochinskiy updated AMQ-855:
-----------------------------------
Attachment: ActiveMQMessageConsumer.patch
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0.2, 4.0.1, 4.0
> Environment: any
> Reporter: Vadim Pesochinskiy
> Priority: Critical
> Fix For: 4.1
>
> Attachments: ActiveMQMessageConsumer.patch, PrefetchSubscription.patch
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (AMQ-855) Add support for prefetchSize = 0
Posted by "Vadim Pesochinskiy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=all ]
Vadim Pesochinskiy updated AMQ-855:
-----------------------------------
Attachment: PrefetchSubscription.java.patch2
Almost resolved :-). There 2 minor but significant changes.
1. pullMessage(). Changed:
prefetchExtension++ to prefetchExtension=1 The problem with increment is that when message is rolled back by client, consumer will call receive*() again and that will make prefetchExtension 2 and the window will be open permanently again, i.e. goes back to push instead of pull. In case of roll back client does not really need to pull, but there is not much harm in it.
2. acknowledge(ConnectionContext, MessageAck). Changed as follows:
line 191:
++ if (getPrefetchSize() != 0) {
prefetchExtension=Math.max(prefetchExtension,index+1);
++ }
What happens here is when standard ack is received window is extended to push more messages, which we do not need in pulling client situation.
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0, 4.0.1, 4.0.2
> Environment: any
> Reporter: Vadim Pesochinskiy
> Assigned To: Hiram Chirino
> Priority: Critical
> Fix For: 4.1
>
> Attachments: ActiveMQMessageConsumer.patch, ActiveMQMessageConsumer.patch2, PrefetchSubscription.java.patch2, PrefetchSubscription.patch, region.QueueSubscription.java.patch, ZeroPrefetchConsumerTest.java.patch
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AMQ-855) Add support for prefetchSize = 0
Posted by "Vadim Pesochinskiy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=comments#action_36894 ]
Vadim Pesochinskiy commented on AMQ-855:
----------------------------------------
OK. I will take a look at it and the unit tests. Thanks.
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0, 4.0.1, 4.0.2
> Environment: any
> Reporter: Vadim Pesochinskiy
> Assigned To: Hiram Chirino
> Priority: Critical
> Fix For: 4.1
>
> Attachments: ActiveMQMessageConsumer.patch, ActiveMQMessageConsumer.patch2, PrefetchSubscription.patch
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AMQ-855) Add support for prefetchSize = 0
Posted by "Vadim Pesochinskiy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=comments#action_36896 ]
Vadim Pesochinskiy commented on AMQ-855:
----------------------------------------
Actually I think it would be better to change getGroupID() and getSecuence() to return null instead of throwing exception.
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0, 4.0.1, 4.0.2
> Environment: any
> Reporter: Vadim Pesochinskiy
> Assigned To: Hiram Chirino
> Priority: Critical
> Fix For: 4.1
>
> Attachments: ActiveMQMessageConsumer.patch, ActiveMQMessageConsumer.patch2, PrefetchSubscription.patch, region.QueueSubscription.java.patch, ZeroPrefetchConsumerTest.java.patch
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AMQ-855) Add support for prefetchSize = 0
Posted by "Vadim Pesochinskiy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=comments#action_36730 ]
Vadim Pesochinskiy commented on AMQ-855:
----------------------------------------
James & Hiram,
I understand that you are trying to keep the scope of AMQ in check and I appreciate this as I believe we will benefit in quality. However, I just do not think I can proceed with using this product, because it does not satisfy my project's requirements. The requirements are not artificial and I am sure they cannot be resolved with either of the solutions that you proposed. I do not want to abuse your CPU with long explanation of why it is so.
Due to whatever reasons I have multiple synchronous consumers (no listener, receive*(*) methods) and some of them are not used for extended periods of time. This results in the message broker not dispatching messages. Is this not a bug?!!!
I do not know if setting prefetchSize=0 is the solution, but somehow AMQ needs to support the case when consumers are not getting any messages until they request them.
- Vadim.
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0, 4.0.1, 4.0.2
> Environment: any
> Reporter: Vadim Pesochinskiy
> Priority: Critical
> Fix For: 4.2
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Assigned: (AMQ-855) Add support for prefetchSize = 0
Posted by "james strachan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=all ]
james strachan reassigned AMQ-855:
----------------------------------
Assignee: Hiram Chirino
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0, 4.0.1, 4.0.2
> Environment: any
> Reporter: Vadim Pesochinskiy
> Assigned To: Hiram Chirino
> Priority: Critical
> Fix For: 4.1
>
> Attachments: ActiveMQMessageConsumer.patch, ActiveMQMessageConsumer.patch2, PrefetchSubscription.patch
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (AMQ-855) Add support for prefetchSize = 0
Posted by "Vadim Pesochinskiy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=all ]
Vadim Pesochinskiy updated AMQ-855:
-----------------------------------
Attachment: PrefetchSubscription.java.patch3
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0, 4.0.1, 4.0.2
> Environment: any
> Reporter: Vadim Pesochinskiy
> Assigned To: Hiram Chirino
> Priority: Critical
> Fix For: 4.1
>
> Attachments: ActiveMQMessageConsumer.java.patch3, ActiveMQMessageConsumer.patch, ActiveMQMessageConsumer.patch2, PrefetchSubscription.java.patch2, PrefetchSubscription.java.patch3, PrefetchSubscription.patch, region.QueueSubscription.java.patch, ZeroPrefetchConsumerTest.java.patch, ZeroPrefetchConsumerTest.java.patch3
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AMQ-855) Add support for prefetchSize = 0
Posted by "Vadim Pesochinskiy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=comments#action_36736 ]
Vadim Pesochinskiy commented on AMQ-855:
----------------------------------------
> e.g. if we did implement prefetch of zero - which means don't deliver a message to a consumer at all -
> unless they perform a consumer.receive() - even then, the consumer could then hang/deadlock and never
> actually acknowledge or process the message.
If consumer hangs and message is not processed is a different problem. Even if messages are lost in such case, nobody would blame AMQ. If you kill consumer and re-despatch, it would be brilliant. But if my consumers do not hang, crash or burn, can I just get my messages?
James, could you please re-open this issue? I spend 1 month working on this project and now I will have to throw it all away and do explaining with my manager. I cannot apply my own fix, you will not fix it. What am I supposed to do now?
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0, 4.0.1, 4.0.2
> Environment: any
> Reporter: Vadim Pesochinskiy
> Priority: Critical
> Fix For: 4.2
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Reopened: (AMQ-855) Add support for prefetchSize = 0
Posted by "Vadim Pesochinskiy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=all ]
Vadim Pesochinskiy reopened AMQ-855:
------------------------------------
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0, 4.0.1, 4.0.2
> Environment: any
> Reporter: Vadim Pesochinskiy
> Assigned To: Hiram Chirino
> Priority: Critical
> Fix For: 4.1
>
> Attachments: ActiveMQMessageConsumer.patch, ActiveMQMessageConsumer.patch2, PrefetchSubscription.java.patch2, PrefetchSubscription.patch, region.QueueSubscription.java.patch, ZeroPrefetchConsumerTest.java.patch
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AMQ-855) Add support for prefetchSize = 0
Posted by "Hiram Chirino (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=comments#action_36892 ]
Hiram Chirino commented on AMQ-855:
-----------------------------------
'm going to start looking into it today. I may not use the
DataResponse to return the message right away. I think I would still
prefer to use the normal message dispatching flow and logic.
What I'm thinking is that on timeout, or when receiveNoWait() is
called, and no message has arrived to be dispatched, we dispatch a
message to the consumer letting it know that no message was available.
Then it can unblock. The nice thing about this is that you don't get
timing issues between the client and server where the client times out
but the broker actually dispatched the message.
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0, 4.0.1, 4.0.2
> Environment: any
> Reporter: Vadim Pesochinskiy
> Assigned To: Hiram Chirino
> Priority: Critical
> Fix For: 4.1
>
> Attachments: ActiveMQMessageConsumer.patch, ActiveMQMessageConsumer.patch2, PrefetchSubscription.patch
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AMQ-855) Add support for prefetchSize = 0
Posted by "Hiram Chirino (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=comments#action_36898 ]
Hiram Chirino commented on AMQ-855:
-----------------------------------
Thanks Vadim,
patch applied in trunk rev 439804.
Test case passes without error.
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0, 4.0.1, 4.0.2
> Environment: any
> Reporter: Vadim Pesochinskiy
> Assigned To: Hiram Chirino
> Priority: Critical
> Fix For: 4.1
>
> Attachments: ActiveMQMessageConsumer.patch, ActiveMQMessageConsumer.patch2, PrefetchSubscription.patch, region.QueueSubscription.java.patch, ZeroPrefetchConsumerTest.java.patch
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AMQ-855) Add support for prefetchSize = 0
Posted by "Andrew Lusk (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-855?page=comments#action_36713 ]
Andrew Lusk commented on AMQ-855:
---------------------------------
Hiram, I don't really understand your comments here. There is a huge need for this. A push model just isn't appropriate for queues. Now, you could in theory hack together a "pull" model by setting prefetch size = 0, so that basically each ack is a request for the next message. This method is slower, but it does scale out as you add more consumers. This is a big problem for our internal customers.
Systems like this need to be designed under the assumption that clients will not behave themselves. They will deadlock themselves, slow down arbitrarily, boxes will go up and down, etc. These things happen all the time in real life and shouldn't have adverse effects on other, well-behaved consumers.
> Add support for prefetchSize = 0
> --------------------------------
>
> Key: AMQ-855
> URL: https://issues.apache.org/activemq/browse/AMQ-855
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Broker
> Affects Versions: 4.0, 4.0.1, 4.0.2
> Environment: any
> Reporter: Vadim Pesochinskiy
> Priority: Critical
> Fix For: 4.2
>
>
> This feature would enable to support following test case:
> 2 servers are processing 3 submitted jobs with following processing times 10 min, 1 min, 1 min. This sequence should finish in 10 minutes as one service will pick up the 10 minutes job, meanwhile the other one should manage the two 1 minute jobs. Since I cannot set prefetchSize=0, one of the 1 minute jobs is sitting in prefetch buffer and the jobs are processed in 11 minutes instead of 10.
> This is simplification of the real scenario where I have about 30 consumers submitting jobs to 20 consumers through AMQ 4.0.1. I have following problems:
> Messages are sitting in prefetch buffer are not available to processors, which results in a lot of idle time.
> Order of processing is random. For some reason Job # 20 is processed after Job # 1500. Since senders are synchronously blocked this can result in time-outs.
> Some requests are real-time, i.e. there is a user waiting, so the system cannot wait, so AMQ-850 does not fix this issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira