You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@servicemix.apache.org by "Ron Gavlin (JIRA)" <ji...@apache.org> on 2009/01/24 05:09:10 UTC
[jira] Created: (SM-1777) smx-jms async consumer needs dynamic
mechanism to throttle message consumption
smx-jms async consumer needs dynamic mechanism to throttle message consumption
------------------------------------------------------------------------------
Key: SM-1777
URL: https://issues.apache.org/activemq/browse/SM-1777
Project: ServiceMix
Issue Type: New Feature
Components: servicemix-jms
Affects Versions: servicemix-jms-2008.01
Reporter: Ron Gavlin
Currently, the async smx-jms:consumer has no mechanism to throttle consumption of incoming messages. This becomes problematic in scenarios where a large backlog of messages exists on the input queue. In this case, it is likely that the smx-jms:consumer will overload the smx nmr with message exchanges resulting in problems. A dynamic throttling mechanism on the async smx-jms consumer is required to avoid this problem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SM-1777) smx-jms async consumer needs dynamic
mechanism to throttle message consumption
Posted by "Ron Gavlin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/SM-1777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ron Gavlin updated SM-1777:
---------------------------
Description:
Currently, the async smx-jms:consumer has no mechanism to throttle consumption of incoming messages. This becomes problematic in scenarios where a large backlog of messages exists on the input queue. In this case, it is likely that the smx-jms:consumer will overload the smx nmr with message exchanges resulting in problems. A dynamic throttling mechanism on the async smx-jms consumer is required to avoid this problem.
One possible implementation might as follows:
First, add a property "maxPendingAsyncExchanges" to the new smx-jms:consumer endpoint. Then add support in the endpoint to track the total number of pending/in-progress exchanges, i.e., the number of exchanges for which no DONE/ERROR has been returned. When this value reaches "maxPendingAsyncExchanges", then the runtime would set the Spring JMS "concurrentConsumers" value to 0. Once the number of "pending" exchanges drops below the "maxPendingAsyncExchanges" threshold, the "concurrentConsumers" value would be increased to a value greater than 0.
This is the type of dynamic throttling I was envisioning. Thoughts?
was:
Currently, the async smx-jms:consumer has no mechanism to throttle consumption of incoming messages. This becomes problematic in scenarios where a large backlog of messages exists on the input queue. In this case, it is likely that the smx-jms:consumer will overload the smx nmr with message exchanges resulting in problems. A dynamic throttling mechanism on the async smx-jms consumer is required to avoid this problem.
One possible implementation might as follows:
First, add a property "maxPendingAsyncExchanges" to the new smx-jms:consumer endpoint. Then add support in the endpoint to track the total number of pending/in-progress exchanges, i.e., the number of exchanges for which no DONE/ERROR has been returned. When the value reaches "maxPendingAsyncExchanges", then the runtime would set the Spring JMS "concurrentConsumers" value to 0. Once the number of "pending" exchanges drops below the "maxPendingAsyncExchanges" threshold, the "concurrentConsumers" value would be increased to a value greater than 0.
This is the type of dynamic throttling I was envisioning. Thoughts?
> smx-jms async consumer needs dynamic mechanism to throttle message consumption
> ------------------------------------------------------------------------------
>
> Key: SM-1777
> URL: https://issues.apache.org/activemq/browse/SM-1777
> Project: ServiceMix
> Issue Type: New Feature
> Components: servicemix-jms
> Affects Versions: servicemix-jms-2008.01
> Reporter: Ron Gavlin
>
> Currently, the async smx-jms:consumer has no mechanism to throttle consumption of incoming messages. This becomes problematic in scenarios where a large backlog of messages exists on the input queue. In this case, it is likely that the smx-jms:consumer will overload the smx nmr with message exchanges resulting in problems. A dynamic throttling mechanism on the async smx-jms consumer is required to avoid this problem.
> One possible implementation might as follows:
> First, add a property "maxPendingAsyncExchanges" to the new smx-jms:consumer endpoint. Then add support in the endpoint to track the total number of pending/in-progress exchanges, i.e., the number of exchanges for which no DONE/ERROR has been returned. When this value reaches "maxPendingAsyncExchanges", then the runtime would set the Spring JMS "concurrentConsumers" value to 0. Once the number of "pending" exchanges drops below the "maxPendingAsyncExchanges" threshold, the "concurrentConsumers" value would be increased to a value greater than 0.
> This is the type of dynamic throttling I was envisioning. Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SMXCOMP-98) smx-jms async consumer needs dynamic
mechanism to throttle message consumption
Posted by "Ron Gavlin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/SMXCOMP-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=51319#action_51319 ]
Ron Gavlin commented on SMXCOMP-98:
-----------------------------------
Greetings,
Apache CXF has completed an initial implementation for dynamically throttling async jms message consumption (see https://issues.apache.org/jira/browse/CXF-2002). What are your thoughts on the approach taken by CXF? Do you think a similar technique would be appropriate for resolving this issue?
BTW, it would seem useful to be able to pause the Spring JMS listener by setting concurrentConsumers to 0. Currently, the concurrentConsumers value must be >= 1. Would it make sense to bring this up with the Spring folks on their forums?
Regards,
/Ron
> smx-jms async consumer needs dynamic mechanism to throttle message consumption
> ------------------------------------------------------------------------------
>
> Key: SMXCOMP-98
> URL: https://issues.apache.org/activemq/browse/SMXCOMP-98
> Project: ServiceMix Components
> Issue Type: New Feature
> Components: servicemix-jms
> Affects Versions: servicemix-jms-2008.01
> Reporter: Ron Gavlin
>
> Currently, the async smx-jms:consumer has no mechanism to throttle consumption of incoming messages. This becomes problematic in scenarios where a large backlog of messages exists on the input queue. In this case, it is likely that the smx-jms:consumer will overload the smx nmr with message exchanges resulting in problems. A dynamic throttling mechanism on the async smx-jms consumer is required to avoid this problem.
> One possible implementation might as follows:
> First, add a property "maxPendingAsyncExchanges" to the new smx-jms:consumer endpoint. Then add support in the endpoint to track the total number of pending/in-progress exchanges, i.e., the number of exchanges for which no DONE/ERROR has been returned. When this value reaches "maxPendingAsyncExchanges", then the runtime would set the Spring JMS "concurrentConsumers" value to 0. Once the number of "pending" exchanges drops below the "maxPendingAsyncExchanges" threshold, the "concurrentConsumers" value would be increased to a value greater than 0.
> This is the type of dynamic throttling I was envisioning. Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SMXCOMP-98) smx-jms async consumer needs dynamic
mechanism to throttle message consumption
Posted by "Ron Gavlin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/SMXCOMP-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=51352#action_51352 ]
Ron Gavlin commented on SMXCOMP-98:
-----------------------------------
Here is some additional implementation-related background discussion on this feature:
<rg> i am looking at techniques for throttling the smx-jms consumer using some form of pause/resume mechanism. Do you think invoking stop()/start() on the Spring JMS DMLC is reasonable for such a task?
<gnodet> it could be
<rg> in this case, i presume we would want to set AcceptMessagesWhileStopping to true, right?
<gnodet> can't we dynamically change the number of consumers instead ?
<gnodet> not sure if it support that
<gnodet> and not sure if you could set it to 0
<rg> The Spring DMLC requires that concurrentConsumers be >= 1
<rg> CXF-2002 is pausing message ingest by setting a bogus message selector...
<rg> however, message selectors are very inefficient for some brokers, especially when there are lots of messages in the queue
<gnodet> we may want to still accept done / errors messages on the jms side, right ?
<gnodet> in such case, completely pausing the DMLC might be bad
<rg> i am strictly talking about the consumer here
<rg> for the in-only scenario
<rg> for in-out meps, yes, we want to stop the "in" but still leave the "out" started...do that make sense?
<gnodet> yes
<rg> for the in-out case, we have two distinct DMLCs, correct?
<rg> so to me, start/stop the DMLC is preferred over the bogus message selector approach...at a high level, would you agree?
<gnodet> i don't have any problems with that
<rg> i think the reason concurrentConsumers must be >= 1 is that if you want to set to zero the expectation is that you would simply stop the container...does that make sense?
<gnodet> maybe, but stopping / starting the container might be more expensive
<gnodet> so we need to take care about not doing that for every message when at the threshold point
<rg> are you proposing we have a configurable window around the "maxPendingExchanges" value to avoid overhead issues incurred at the threshold point?
<gnodet> yeah, that might be a good idea
<gnodet> so that we don't stop/restat the DMLC each time a message has been handled
> smx-jms async consumer needs dynamic mechanism to throttle message consumption
> ------------------------------------------------------------------------------
>
> Key: SMXCOMP-98
> URL: https://issues.apache.org/activemq/browse/SMXCOMP-98
> Project: ServiceMix Components
> Issue Type: New Feature
> Components: servicemix-jms
> Affects Versions: servicemix-jms-2008.01
> Reporter: Ron Gavlin
>
> Currently, the async smx-jms:consumer has no mechanism to throttle consumption of incoming messages. This becomes problematic in scenarios where a large backlog of messages exists on the input queue. In this case, it is likely that the smx-jms:consumer will overload the smx nmr with message exchanges resulting in problems. A dynamic throttling mechanism on the async smx-jms consumer is required to avoid this problem.
> One possible implementation might as follows:
> First, add a property "maxPendingAsyncExchanges" to the new smx-jms:consumer endpoint. Then add support in the endpoint to track the total number of pending/in-progress exchanges, i.e., the number of exchanges for which no DONE/ERROR has been returned. When this value reaches "maxPendingAsyncExchanges", then the runtime would set the Spring JMS "concurrentConsumers" value to 0. Once the number of "pending" exchanges drops below the "maxPendingAsyncExchanges" threshold, the "concurrentConsumers" value would be increased to a value greater than 0.
> This is the type of dynamic throttling I was envisioning. Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SM-1777) smx-jms async consumer needs dynamic
mechanism to throttle message consumption
Posted by "Ron Gavlin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/SM-1777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ron Gavlin updated SM-1777:
---------------------------
Description:
Currently, the async smx-jms:consumer has no mechanism to throttle consumption of incoming messages. This becomes problematic in scenarios where a large backlog of messages exists on the input queue. In this case, it is likely that the smx-jms:consumer will overload the smx nmr with message exchanges resulting in problems. A dynamic throttling mechanism on the async smx-jms consumer is required to avoid this problem.
One possible implementation might as follows:
First, add a property "maxPendingAsyncExchanges" to the new smx-jms:consumer endpoint. Then add support in the endpoint to track the total number of pending/in-progress exchanges, i.e., the number of exchanges for which no DONE/ERROR has been returned. When the value reaches "maxPendingAsyncExchanges", then the runtime would set the Spring JMS "concurrentConsumers" value to 0. Once the number of "pending" exchanges drops below the "maxPendingAsyncExchanges" threshold, the "concurrentConsumers" value would be increased to a value greater than 0.
This is the type of dynamic throttling I was envisioning. Thoughts?
was:Currently, the async smx-jms:consumer has no mechanism to throttle consumption of incoming messages. This becomes problematic in scenarios where a large backlog of messages exists on the input queue. In this case, it is likely that the smx-jms:consumer will overload the smx nmr with message exchanges resulting in problems. A dynamic throttling mechanism on the async smx-jms consumer is required to avoid this problem.
> smx-jms async consumer needs dynamic mechanism to throttle message consumption
> ------------------------------------------------------------------------------
>
> Key: SM-1777
> URL: https://issues.apache.org/activemq/browse/SM-1777
> Project: ServiceMix
> Issue Type: New Feature
> Components: servicemix-jms
> Affects Versions: servicemix-jms-2008.01
> Reporter: Ron Gavlin
>
> Currently, the async smx-jms:consumer has no mechanism to throttle consumption of incoming messages. This becomes problematic in scenarios where a large backlog of messages exists on the input queue. In this case, it is likely that the smx-jms:consumer will overload the smx nmr with message exchanges resulting in problems. A dynamic throttling mechanism on the async smx-jms consumer is required to avoid this problem.
> One possible implementation might as follows:
> First, add a property "maxPendingAsyncExchanges" to the new smx-jms:consumer endpoint. Then add support in the endpoint to track the total number of pending/in-progress exchanges, i.e., the number of exchanges for which no DONE/ERROR has been returned. When the value reaches "maxPendingAsyncExchanges", then the runtime would set the Spring JMS "concurrentConsumers" value to 0. Once the number of "pending" exchanges drops below the "maxPendingAsyncExchanges" threshold, the "concurrentConsumers" value would be increased to a value greater than 0.
> This is the type of dynamic throttling I was envisioning. Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.