You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@uima.apache.org by "Eddie Epstein (JIRA)" <ui...@incubator.apache.org> on 2008/07/12 14:47:31 UTC
[jira] Created: (UIMA-1110) The default setting of prefetch=1 is
causing problems
The default setting of prefetch=1 is causing problems
-----------------------------------------------------
Key: UIMA-1110
URL: https://issues.apache.org/jira/browse/UIMA-1110
Project: UIMA
Issue Type: Bug
Components: Async Scaleout
Reporter: Eddie Epstein
The new JMX monitor occasionally shows idle times of many seconds for a scaled out remote delegate that is getting many CASes per second. Assuming the monitor is correct, the problem is that AMQ is not evenly distributing the work to all the service instances. Other recent experiences have demonstrated that uneven distribution is solved with prefetch=0.
While there is some possibility that prefetch > 0 may be useful in some situations for remote delegates, for colocated delegates there is little doubt that a prefetch greater than 0 cannot help performance and can only screw up load balancing.
prefetch=0 should be the default everywhere.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (UIMA-1110) The default setting of prefetch=1 is
causing problems
Posted by "Jerry Cwiklik (JIRA)" <ui...@incubator.apache.org>.
[ https://issues.apache.org/jira/browse/UIMA-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jerry Cwiklik updated UIMA-1110:
--------------------------------
Attachment: uimaj-as-activemq-UIMA-1110-patch.txt
Created a dedicated ConnectionFactory for temp queues that handle CAS Release Requests. Set a different prefetch policy for this factory that uses 10 for a prefetch value.
> The default setting of prefetch=1 is causing problems
> -----------------------------------------------------
>
> Key: UIMA-1110
> URL: https://issues.apache.org/jira/browse/UIMA-1110
> Project: UIMA
> Issue Type: Bug
> Components: Async Scaleout
> Reporter: Eddie Epstein
> Assignee: Eddie Epstein
> Attachments: uimaj-as-activemq-UIMA-1110-patch.txt
>
>
> The new JMX monitor occasionally shows idle times of many seconds for a scaled out remote delegate that is getting many CASes per second. Assuming the monitor is correct, the problem is that AMQ is not evenly distributing the work to all the service instances. Other recent experiences have demonstrated that uneven distribution is solved with prefetch=0.
> While there is some possibility that prefetch > 0 may be useful in some situations for remote delegates, for colocated delegates there is little doubt that a prefetch greater than 0 cannot help performance and can only screw up load balancing.
> prefetch=0 should be the default everywhere.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (UIMA-1110) The default setting of prefetch=1 is
causing problems
Posted by "Eddie Epstein (JIRA)" <ui...@incubator.apache.org>.
[ https://issues.apache.org/jira/browse/UIMA-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662478#action_12662478 ]
Eddie Epstein commented on UIMA-1110:
-------------------------------------
This situation appears to be resolved.
1. the default prefetch value for a service input queue is 0, which results in accurate load balancing. If there is a bandwidth problem with 0, which we have not seen yet, it can be overridden in the deployment descriptor.
2. remote delegates now always have a [temp] reply queue on the remote broker, and the prefetch value for these are always 1. Testing has shown a big improvement going from 0 to 1, but little or no difference between 1 and 10. Values larger than 1 can be problematic when reply CASes are very large.
3. the free CAS queue should always use a value of prefetch>0. We need to confirm this is done.
> The default setting of prefetch=1 is causing problems
> -----------------------------------------------------
>
> Key: UIMA-1110
> URL: https://issues.apache.org/jira/browse/UIMA-1110
> Project: UIMA
> Issue Type: Bug
> Components: Async Scaleout
> Reporter: Eddie Epstein
> Assignee: Eddie Epstein
>
> The new JMX monitor occasionally shows idle times of many seconds for a scaled out remote delegate that is getting many CASes per second. Assuming the monitor is correct, the problem is that AMQ is not evenly distributing the work to all the service instances. Other recent experiences have demonstrated that uneven distribution is solved with prefetch=0.
> While there is some possibility that prefetch > 0 may be useful in some situations for remote delegates, for colocated delegates there is little doubt that a prefetch greater than 0 cannot help performance and can only screw up load balancing.
> prefetch=0 should be the default everywhere.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (UIMA-1110) The default setting of prefetch=1 is
causing problems
Posted by "Jerry Cwiklik (JIRA)" <ui...@incubator.apache.org>.
[ https://issues.apache.org/jira/browse/UIMA-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jerry Cwiklik closed UIMA-1110.
-------------------------------
Resolution: Fixed
> The default setting of prefetch=1 is causing problems
> -----------------------------------------------------
>
> Key: UIMA-1110
> URL: https://issues.apache.org/jira/browse/UIMA-1110
> Project: UIMA
> Issue Type: Bug
> Components: Async Scaleout
> Reporter: Eddie Epstein
> Assignee: Eddie Epstein
> Attachments: uimaj-as-activemq-UIMA-1110-patch.txt
>
>
> The new JMX monitor occasionally shows idle times of many seconds for a scaled out remote delegate that is getting many CASes per second. Assuming the monitor is correct, the problem is that AMQ is not evenly distributing the work to all the service instances. Other recent experiences have demonstrated that uneven distribution is solved with prefetch=0.
> While there is some possibility that prefetch > 0 may be useful in some situations for remote delegates, for colocated delegates there is little doubt that a prefetch greater than 0 cannot help performance and can only screw up load balancing.
> prefetch=0 should be the default everywhere.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (UIMA-1110) The default setting of prefetch=1 is
causing problems
Posted by "Eddie Epstein (JIRA)" <ui...@incubator.apache.org>.
[ https://issues.apache.org/jira/browse/UIMA-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eddie Epstein reopened UIMA-1110:
---------------------------------
After discussions with Jerry and Marshall, the situation with prefetch is more complicated and needs more work.
In the current design, a single prefetch policy is used for all queues defined by a UIMA AS service. For an AS primitive these would be the input queue and, if a CAS multiplier, the free CAS queue. For an AS aggregate, these would be the top-level input queue, remote reply queues, free CAS queues and all queues defined on the internal broker used for colocated components.
The intention of the prefetch option in the deployment descriptor was for users to be able to customize only the top-level service input queue. However, it is used to set the global prefetch policy described above.
I propose one of the the following alternatives:
Alternative 1:
- set prefetch=0 for all queues (minimum complexity)
Alternative 2:
- set prefetch=0 for all queues, except
- the free CAS queue for a CAS multiplier, prefetch = 10 (messages always small)
Alternative 3: allow customization of the following:
- top-level input queue, default = 0
- reply queue for each remote delegate, default = 0
- the free CAS queue for a CAS multiplier service, default = 10
> The default setting of prefetch=1 is causing problems
> -----------------------------------------------------
>
> Key: UIMA-1110
> URL: https://issues.apache.org/jira/browse/UIMA-1110
> Project: UIMA
> Issue Type: Bug
> Components: Async Scaleout
> Reporter: Eddie Epstein
> Assignee: Eddie Epstein
>
> The new JMX monitor occasionally shows idle times of many seconds for a scaled out remote delegate that is getting many CASes per second. Assuming the monitor is correct, the problem is that AMQ is not evenly distributing the work to all the service instances. Other recent experiences have demonstrated that uneven distribution is solved with prefetch=0.
> While there is some possibility that prefetch > 0 may be useful in some situations for remote delegates, for colocated delegates there is little doubt that a prefetch greater than 0 cannot help performance and can only screw up load balancing.
> prefetch=0 should be the default everywhere.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (UIMA-1110) The default setting of prefetch=1 is
causing problems
Posted by "Marshall Schor (JIRA)" <ui...@incubator.apache.org>.
[ https://issues.apache.org/jira/browse/UIMA-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Marshall Schor resolved UIMA-1110.
----------------------------------
Resolution: Fixed
Assignee: Eddie Epstein
changed default value of prefetch to 0 if not specified, in all cases including co-located.
> The default setting of prefetch=1 is causing problems
> -----------------------------------------------------
>
> Key: UIMA-1110
> URL: https://issues.apache.org/jira/browse/UIMA-1110
> Project: UIMA
> Issue Type: Bug
> Components: Async Scaleout
> Reporter: Eddie Epstein
> Assignee: Eddie Epstein
>
> The new JMX monitor occasionally shows idle times of many seconds for a scaled out remote delegate that is getting many CASes per second. Assuming the monitor is correct, the problem is that AMQ is not evenly distributing the work to all the service instances. Other recent experiences have demonstrated that uneven distribution is solved with prefetch=0.
> While there is some possibility that prefetch > 0 may be useful in some situations for remote delegates, for colocated delegates there is little doubt that a prefetch greater than 0 cannot help performance and can only screw up load balancing.
> prefetch=0 should be the default everywhere.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (UIMA-1110) The default setting of prefetch=1 is
causing problems
Posted by "Marshall Schor (JIRA)" <ui...@incubator.apache.org>.
[ https://issues.apache.org/jira/browse/UIMA-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12626322#action_12626322 ]
Marshall Schor commented on UIMA-1110:
--------------------------------------
Also, see UIMA-1059 - which was closed with won't fix, to be considered with this Jira instead. (Issue in 1059 was to update dd2spring to disallow setting prefetch on inner aggregates).
If we go with alternative 3, which is the one suggesting prefetch should be associated with remote delegates, I think the current syntax for prefetch (which is to put it as an attribute on the inputQueue element) is confusing - since it has nothing here to do with the remote delegate's input queue. Instead, it could be treated more like we did the additional tuning attributes for remoteReplyQueueScaleout.
I agree the top level prefetch should not be used to set the prefetch for everything. The prefetch for the free CAS Q for a cas multiplier = 10 is a good example.
What about Alternative 4: setting prefetch for the top level queue via the deployment descriptor arg, but then having the the others work as Alternative 3? As I recall, the remoteReplyQueueScaleout - which could be used to make more threads available for reading the reply q - was thought to be an alternative to setting the prefetch for the remote reply queue above 0.
> The default setting of prefetch=1 is causing problems
> -----------------------------------------------------
>
> Key: UIMA-1110
> URL: https://issues.apache.org/jira/browse/UIMA-1110
> Project: UIMA
> Issue Type: Bug
> Components: Async Scaleout
> Reporter: Eddie Epstein
> Assignee: Eddie Epstein
>
> The new JMX monitor occasionally shows idle times of many seconds for a scaled out remote delegate that is getting many CASes per second. Assuming the monitor is correct, the problem is that AMQ is not evenly distributing the work to all the service instances. Other recent experiences have demonstrated that uneven distribution is solved with prefetch=0.
> While there is some possibility that prefetch > 0 may be useful in some situations for remote delegates, for colocated delegates there is little doubt that a prefetch greater than 0 cannot help performance and can only screw up load balancing.
> prefetch=0 should be the default everywhere.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.