You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@uima.apache.org by "Jerry Cwiklik (JIRA)" <de...@uima.apache.org> on 2010/10/15 16:19:32 UTC
[jira] Created: (UIMA-1902) UIMA AS should send an ACK message to a
client for each request
UIMA AS should send an ACK message to a client for each request
---------------------------------------------------------------
Key: UIMA-1902
URL: https://issues.apache.org/jira/browse/UIMA-1902
Project: UIMA
Issue Type: Bug
Components: Async Scaleout
Affects Versions: 2.3AS
Reporter: Jerry Cwiklik
Assignee: Jerry Cwiklik
UIMA AS service checks for existence of a reply queue before processing an incoming request. This is done to prevent wasting cycles processing a request if it is known that the client has gone away. Current approach is based on checking broker's queue inventory for presence of a temp reply queue provided in a request msg. The framework code uses JMX connection to a broker to perform the queue lookup. If the lookup fails, the framework concludes that the client has gone away, places the reply queue in a list of destinations known to be dead and throws away the request msg. Any subsequent msg containing the same reply queue is automatically thrown away since the destination is marked as dead. Occasionally, the JMX lookup fails even though (as reported by some users) the client is alive at the time the lookup was done. It seems that perhaps under load JMX fails apart, reporting false positive result ( queue not present). The failed lookup results in a dropped request msg and ultimately in a client timeout (or possible hang if the client is not using timout).
Modify the implementation to stop using JMX to perform lookups. Instead, before any processing takes place, send a small msg (ACK) back to the client via JMS. If the send fails, report the problem, drop the request and wait for the next request. Also, dont add failed destination to the list of clients known to be dead. Each request should be subject to an ACK no matter what happened previously.
The new approach, although generating more jms traffic, has some advantages:
1) A client is immediately notified when its request has been picked by a service and is ready to be processed.
2) A client "knows" which service is processing each request
3) A client can use a different timeout strategy since it knows precisely when its request is ready to be processed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (UIMA-1902) UIMA AS detection of missing client
reply queue is not working leading to client timeouts
Posted by "Jerry Cwiklik (JIRA)" <de...@uima.apache.org>.
[ https://issues.apache.org/jira/browse/UIMA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jerry Cwiklik updated UIMA-1902:
--------------------------------
Description:
UIMA AS service checks for existence of a reply queue before processing an incoming request. This is done to prevent wasting cycles processing a request if it is known that the client has gone away. Current approach is based on checking broker's queue inventory for presence of a temp reply queue provided in a request msg. The framework code uses JMX connection to a broker to perform the queue lookup. If the lookup fails, the framework concludes that the client has gone away, places the reply queue in a list of destinations known to be dead and throws away the request msg. Any subsequent msg containing the same reply queue is automatically thrown away since the destination is marked as dead. Occasionally, the JMX lookup fails even though (as reported by some users) the client is alive at the time the lookup was done. It seems that perhaps under load JMX fails, reporting false positive result ( queue not present). The failed lookup results in a dropped request msg and ultimately in a client timeout (or possible hang if the client is not using timout).
Modify the implementation to stop using JMX to perform lookups. Instead, before any processing takes place, send a small msg (ACK) back to the client via JMS. If the send fails, report the problem, drop the request and wait for the next request. Also, dont add failed destination to the list of clients known to be dead. Each request should be subject to an ACK no matter what happened previously.
The new approach, although generating more jms traffic, has some advantages:
1) A client is immediately notified when its request has been picked by a service and is ready to be processed.
2) A client "knows" which service is processing each request
3) A client can use a different timeout strategy since it knows precisely when its request is ready to be processed.
was:
UIMA AS service checks for existence of a reply queue before processing an incoming request. This is done to prevent wasting cycles processing a request if it is known that the client has gone away. Current approach is based on checking broker's queue inventory for presence of a temp reply queue provided in a request msg. The framework code uses JMX connection to a broker to perform the queue lookup. If the lookup fails, the framework concludes that the client has gone away, places the reply queue in a list of destinations known to be dead and throws away the request msg. Any subsequent msg containing the same reply queue is automatically thrown away since the destination is marked as dead. Occasionally, the JMX lookup fails even though (as reported by some users) the client is alive at the time the lookup was done. It seems that perhaps under load JMX fails apart, reporting false positive result ( queue not present). The failed lookup results in a dropped request msg and ultimately in a client timeout (or possible hang if the client is not using timout).
Modify the implementation to stop using JMX to perform lookups. Instead, before any processing takes place, send a small msg (ACK) back to the client via JMS. If the send fails, report the problem, drop the request and wait for the next request. Also, dont add failed destination to the list of clients known to be dead. Each request should be subject to an ACK no matter what happened previously.
The new approach, although generating more jms traffic, has some advantages:
1) A client is immediately notified when its request has been picked by a service and is ready to be processed.
2) A client "knows" which service is processing each request
3) A client can use a different timeout strategy since it knows precisely when its request is ready to be processed.
> UIMA AS detection of missing client reply queue is not working leading to client timeouts
> -----------------------------------------------------------------------------------------
>
> Key: UIMA-1902
> URL: https://issues.apache.org/jira/browse/UIMA-1902
> Project: UIMA
> Issue Type: Bug
> Components: Async Scaleout
> Affects Versions: 2.3AS
> Reporter: Jerry Cwiklik
> Assignee: Jerry Cwiklik
>
> UIMA AS service checks for existence of a reply queue before processing an incoming request. This is done to prevent wasting cycles processing a request if it is known that the client has gone away. Current approach is based on checking broker's queue inventory for presence of a temp reply queue provided in a request msg. The framework code uses JMX connection to a broker to perform the queue lookup. If the lookup fails, the framework concludes that the client has gone away, places the reply queue in a list of destinations known to be dead and throws away the request msg. Any subsequent msg containing the same reply queue is automatically thrown away since the destination is marked as dead. Occasionally, the JMX lookup fails even though (as reported by some users) the client is alive at the time the lookup was done. It seems that perhaps under load JMX fails, reporting false positive result ( queue not present). The failed lookup results in a dropped request msg and ultimately in a client timeout (or possible hang if the client is not using timout).
> Modify the implementation to stop using JMX to perform lookups. Instead, before any processing takes place, send a small msg (ACK) back to the client via JMS. If the send fails, report the problem, drop the request and wait for the next request. Also, dont add failed destination to the list of clients known to be dead. Each request should be subject to an ACK no matter what happened previously.
> The new approach, although generating more jms traffic, has some advantages:
> 1) A client is immediately notified when its request has been picked by a service and is ready to be processed.
> 2) A client "knows" which service is processing each request
> 3) A client can use a different timeout strategy since it knows precisely when its request is ready to be processed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (UIMA-1902) UIMA AS detection of missing client
reply queue is not working leading to client timeouts
Posted by "Jerry Cwiklik (JIRA)" <de...@uima.apache.org>.
[ https://issues.apache.org/jira/browse/UIMA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jerry Cwiklik updated UIMA-1902:
--------------------------------
Summary: UIMA AS detection of missing client reply queue is not working leading to client timeouts (was: UIMA AS should send an ACK message to a client for each request)
> UIMA AS detection of missing client reply queue is not working leading to client timeouts
> -----------------------------------------------------------------------------------------
>
> Key: UIMA-1902
> URL: https://issues.apache.org/jira/browse/UIMA-1902
> Project: UIMA
> Issue Type: Bug
> Components: Async Scaleout
> Affects Versions: 2.3AS
> Reporter: Jerry Cwiklik
> Assignee: Jerry Cwiklik
>
> UIMA AS service checks for existence of a reply queue before processing an incoming request. This is done to prevent wasting cycles processing a request if it is known that the client has gone away. Current approach is based on checking broker's queue inventory for presence of a temp reply queue provided in a request msg. The framework code uses JMX connection to a broker to perform the queue lookup. If the lookup fails, the framework concludes that the client has gone away, places the reply queue in a list of destinations known to be dead and throws away the request msg. Any subsequent msg containing the same reply queue is automatically thrown away since the destination is marked as dead. Occasionally, the JMX lookup fails even though (as reported by some users) the client is alive at the time the lookup was done. It seems that perhaps under load JMX fails apart, reporting false positive result ( queue not present). The failed lookup results in a dropped request msg and ultimately in a client timeout (or possible hang if the client is not using timout).
> Modify the implementation to stop using JMX to perform lookups. Instead, before any processing takes place, send a small msg (ACK) back to the client via JMS. If the send fails, report the problem, drop the request and wait for the next request. Also, dont add failed destination to the list of clients known to be dead. Each request should be subject to an ACK no matter what happened previously.
> The new approach, although generating more jms traffic, has some advantages:
> 1) A client is immediately notified when its request has been picked by a service and is ready to be processed.
> 2) A client "knows" which service is processing each request
> 3) A client can use a different timeout strategy since it knows precisely when its request is ready to be processed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.