You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@servicemix.apache.org by "Ron Gavlin (JIRA)" <ji...@apache.org> on 2009/02/05 12:38:59 UTC
[jira] Created: (SM-1789) smx-file async FilePollerEndpoint needs a
throttling mechanism to avoid creating excessive numbers of open exchanges
and overloading the nmr
smx-file async FilePollerEndpoint needs a throttling mechanism to avoid creating excessive numbers of open exchanges and overloading the nmr
--------------------------------------------------------------------------------------------------------------------------------------------
Key: SM-1789
URL: https://issues.apache.org/activemq/browse/SM-1789
Project: ServiceMix
Issue Type: Bug
Components: servicemix-jms
Affects Versions: servicemix-file-2008.01
Reporter: Ron Gavlin
Priority: Critical
now that the smx-file poller is asynchronous, i run into saturation problems when several thousand files exist in the polled directory at startup. This is similar to the problems I encounter in the async smx-jms consumer when I have thousands of messages on the queue at startup. The jms issue is captured in SM-1777.
I suggest that an abstract throttling mechanism be implemented so that config options can be shared across components. I suspect the primary config option would be maxOpenExchanges. When this value is exceeded for the smx-file poller, it would stop polling until the maxOpenExchanges is reduced below maxOpenExchange. For the smx-jms consumer or the cxf-bc jms consumer, setting the Spring JMS concurrentConsumers to 0 would stop consumption.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SM-1789) smx-file async FilePollerEndpoint needs
a throttling mechanism to avoid creating excessive numbers of open
exchanges and overloading the nmr
Posted by "Ron Gavlin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/SM-1789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=49437#action_49437 ]
Ron Gavlin commented on SM-1789:
--------------------------------
Chris,
You might want to take a look at the throttling technique Guillaume is using in the new SMX4 cluster endpoint and follow suit when solving this problem. I believe he is using property maxPendingExchanges to capture the throttling high-water mark.
> smx-file async FilePollerEndpoint needs a throttling mechanism to avoid creating excessive numbers of open exchanges and overloading the nmr
> --------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SM-1789
> URL: https://issues.apache.org/activemq/browse/SM-1789
> Project: ServiceMix
> Issue Type: Bug
> Components: servicemix-file
> Affects Versions: servicemix-file-2008.01
> Reporter: Ron Gavlin
> Assignee: Chris Custine
> Priority: Critical
>
> now that the smx-file poller is asynchronous, i run into saturation problems when several thousand files exist in the polled directory at startup. This is similar to the problems I encounter in the async smx-jms consumer when I have thousands of messages on the queue at startup. The jms issue is captured in SM-1777.
> I suggest that an abstract throttling mechanism be implemented so that config options can be shared across components. I suspect the primary config option would be maxOpenExchanges. When this value is exceeded for the smx-file poller, it would stop polling until the maxOpenExchanges is reduced below maxOpenExchange. For the smx-jms consumer or the cxf-bc jms consumer, setting the Spring JMS concurrentConsumers to 0 would stop consumption.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SM-1789) smx-file async FilePollerEndpoint needs a
throttling mechanism to avoid creating excessive numbers of open exchanges
and overloading the nmr
Posted by "Chris Custine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/SM-1789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Custine updated SM-1789:
------------------------------
Component/s: (was: servicemix-jms)
servicemix-file
Assignee: Chris Custine
> smx-file async FilePollerEndpoint needs a throttling mechanism to avoid creating excessive numbers of open exchanges and overloading the nmr
> --------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SM-1789
> URL: https://issues.apache.org/activemq/browse/SM-1789
> Project: ServiceMix
> Issue Type: Bug
> Components: servicemix-file
> Affects Versions: servicemix-file-2008.01
> Reporter: Ron Gavlin
> Assignee: Chris Custine
> Priority: Critical
>
> now that the smx-file poller is asynchronous, i run into saturation problems when several thousand files exist in the polled directory at startup. This is similar to the problems I encounter in the async smx-jms consumer when I have thousands of messages on the queue at startup. The jms issue is captured in SM-1777.
> I suggest that an abstract throttling mechanism be implemented so that config options can be shared across components. I suspect the primary config option would be maxOpenExchanges. When this value is exceeded for the smx-file poller, it would stop polling until the maxOpenExchanges is reduced below maxOpenExchange. For the smx-jms consumer or the cxf-bc jms consumer, setting the Spring JMS concurrentConsumers to 0 would stop consumption.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (SMXCOMP-52) smx-file async FilePollerEndpoint
needs a throttling mechanism to avoid creating excessive numbers of open
exchanges and overloading the nmr
Posted by "Chris Custine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/SMXCOMP-52?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Custine resolved SMXCOMP-52.
----------------------------------
Resolution: Fixed
Fix Version/s: servicemix-file-2009.01
> smx-file async FilePollerEndpoint needs a throttling mechanism to avoid creating excessive numbers of open exchanges and overloading the nmr
> --------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SMXCOMP-52
> URL: https://issues.apache.org/activemq/browse/SMXCOMP-52
> Project: ServiceMix Components
> Issue Type: Bug
> Components: servicemix-file
> Affects Versions: servicemix-file-2008.01
> Reporter: Ron Gavlin
> Assignee: Chris Custine
> Priority: Critical
> Fix For: servicemix-file-2009.01
>
>
> now that the smx-file poller is asynchronous, i run into saturation problems when several thousand files exist in the polled directory at startup. This is similar to the problems I encounter in the async smx-jms consumer when I have thousands of messages on the queue at startup. The jms issue is captured in SM-1777.
> I suggest that an abstract throttling mechanism be implemented so that config options can be shared across components. I suspect the primary config option would be maxOpenExchanges. When this value is exceeded for the smx-file poller, it would stop polling until the maxOpenExchanges is reduced below maxOpenExchange. For the smx-jms consumer or the cxf-bc jms consumer, setting the Spring JMS concurrentConsumers to 0 would stop consumption.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SMXCOMP-52) smx-file async FilePollerEndpoint
needs a throttling mechanism to avoid creating excessive numbers of open
exchanges and overloading the nmr
Posted by "Chris Custine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/SMXCOMP-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=49761#action_49761 ]
Chris Custine commented on SMXCOMP-52:
--------------------------------------
Added a property named maxConcurrency to the endpoint configuration. When set to > 0 this will limit the number of pending exchanges when sent async. Since one polling cycle will submit as many processing threads as there are files at that time, this throttle also throttles the submission of processing threads so that the threadpool is not overloaded.
With thousands of files and throttling turned on, I recommend using the default 5000ms or higher for the polling "period".
> smx-file async FilePollerEndpoint needs a throttling mechanism to avoid creating excessive numbers of open exchanges and overloading the nmr
> --------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SMXCOMP-52
> URL: https://issues.apache.org/activemq/browse/SMXCOMP-52
> Project: ServiceMix Components
> Issue Type: Bug
> Components: servicemix-file
> Affects Versions: servicemix-file-2008.01
> Reporter: Ron Gavlin
> Assignee: Chris Custine
> Priority: Critical
> Fix For: servicemix-file-2009.01
>
>
> now that the smx-file poller is asynchronous, i run into saturation problems when several thousand files exist in the polled directory at startup. This is similar to the problems I encounter in the async smx-jms consumer when I have thousands of messages on the queue at startup. The jms issue is captured in SM-1777.
> I suggest that an abstract throttling mechanism be implemented so that config options can be shared across components. I suspect the primary config option would be maxOpenExchanges. When this value is exceeded for the smx-file poller, it would stop polling until the maxOpenExchanges is reduced below maxOpenExchange. For the smx-jms consumer or the cxf-bc jms consumer, setting the Spring JMS concurrentConsumers to 0 would stop consumption.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.