[jira] [Commented] (UIMA-4210) Client hangs with more than 1 time-out

["Jerry Cwiklik (JIRA)" <de...@uima.apache.org>]

    [ https://issues.apache.org/jira/browse/UIMA-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15484937#comment-15484937 ] 

Jerry Cwiklik commented on UIMA-4210:
-------------------------------------

I have instrumented junit test (testPrimitiveServiceProcessPingFailure) to force the scenario described. Not seeing a hang when the code in question is run. Specifically, added tracing in a block of code that was referenced in the problem report (BaseUIMAAsynchronousEngineCommon_impl.notifyOnTimeout())

    	        System.out.println("isSynchronousCall="+isSynchronousCall+" serviceDelegate.getCasPendingReplyListSize()="+serviceDelegate.getCasPendingReplyListSize());
    	        if ( !isSynchronousCall && serviceDelegate.getCasPendingReplyListSize() > 0) {
    	            String nextOutstandingCasReferenceId = 
    	            		serviceDelegate.getOldestCasIdFromOutstandingList();
    	        	if ( nextOutstandingCasReferenceId != null ) {
    	        		cachedRequest = (ClientRequest) clientCache.get(nextOutstandingCasReferenceId);
    	        		if ( cachedRequest != null && cachedRequest.getCAS() != null ) {
        	        		try {
        	        			System.out.println("Sending CAS Again");
        	            		        sendCAS(cachedRequest.getCAS());
        	        		} catch( Exception e) {
        	        			UIMAFramework.getLogger(CLASS_NAME).logrb(Level.WARNING, getClass().getName(),
        	                            "notifyOnTimout", UIMAEE_Constants.JMS_LOG_RESOURCE_BUNDLE,
        	                            "UIMAEE_exception__WARNING", e);
        	        		}
    	        		}
    	        	}
    	        }

When I ran a junit test I could see the trace msgs showing each time a timeout occurred. I can see the sendCAS() is being called and no hang has been observed. 

org.apache.uima.aae.error.UimaASProcessCasTimeout: UIMA AS Client Timed Out Waiting For CAS:-e5dc9fa:1571f849883:-55c1 Reply From a Service On Queue:PersonTitleAnnotatorQueue
runTest: Received Reply from CAS -e5dc9fa:1571f849883:-55c1 Containing 1 Exception(s)
	at org.apache.uima.adapter.jms.client.BaseUIMAAsynchronousEngineCommon_impl.notifyOnTimout(BaseUIMAAsynchronousEngineCommon_impl.java:2372)
	at org.apache.uima.adapter.jms.client.ClientServiceDelegate.handleError(ClientServiceDelegate.java:144)
	at org.apache.uima.aae.delegate.Delegate$1.run(Delegate.java:790)
	at java.util.TimerThread.mainLoop(Timer.java:566)
	at java.util.TimerThread.run(Timer.java:516)
isSynchronousCall=false serviceDelegate.getCasPendingReplyListSize()=3
Sending CAS Again
runTest: Received Reply from CAS -e5dc9fa:1571f849883:-55c0 Containing 1 Exception(s)
org.apache.uima.aae.error.UimaASProcessCasTimeout: UIMA AS Client Timed Out Waiting For CAS:-e5dc9fa:1571f849883:-55c0 Reply From a Service On Queue:PersonTitleAnnotatorQueue
	at org.apache.uima.adapter.jms.client.BaseUIMAAsynchronousEngineCommon_impl.notifyOnTimout(BaseUIMAAsynchronousEngineCommon_impl.java:2372)
	at org.apache.uima.adapter.jms.client.ClientServiceDelegate.handleError(ClientServiceDelegate.java:144)
	at org.apache.uima.aae.delegate.Delegate$1.run(Delegate.java:790)
	at java.util.TimerThread.mainLoop(Timer.java:566)
	at java.util.TimerThread.run(Timer.java:516)
isSynchronousCall=false serviceDelegate.getCasPendingReplyListSize()=2
Sending CAS Again
runTest: Received Reply from CAS -e5dc9fa:1571f849883:-55bf Containing 1 Exception(s)
org.apache.uima.aae.error.UimaASProcessCasTimeout: UIMA AS Client Timed Out Waiting For CAS:-e5dc9fa:1571f849883:-55bf Reply From a Service On Queue:PersonTitleAnnotatorQueue
	at org.apache.uima.adapter.jms.client.BaseUIMAAsynchronousEngineCommon_impl.notifyOnTimout(BaseUIMAAsynchronousEngineCommon_impl.java:2372)
	at org.apache.uima.adapter.jms.client.ClientServiceDelegate.handleError(ClientServiceDelegate.java:144)
	at org.apache.uima.aae.delegate.Delegate$1.run(Delegate.java:790)
	at java.util.TimerThread.mainLoop(Timer.java:566)
	at java.util.TimerThread.run(Timer.java:516)
isSynchronousCall=false serviceDelegate.getCasPendingReplyListSize()=1
Sending CAS Again
runTest: Received Reply from CAS -e5dc9fa:1571f849883:-55bd Containing 1 Exception(s)
runTest: Ping Timeout - service Not Responding To Ping
org.apache.uima.aae.error.UimaASProcessCasTimeout: Service Not Responding to Ping - CAS:-e5dc9fa:1571f849883:-55bd
	at org.apache.uima.adapter.jms.client.ClientServiceDelegate.handleError(ClientServiceDelegate.java:175)
	at org.apache.uima.aae.delegate.Delegate$2.run(Delegate.java:835)
	at java.util.TimerThread.mainLoop(Timer.java:566)
	at java.util.TimerThread.run(Timer.java:516)
Caused by: org.apache.uima.aae.error.UimaASPingTimeout: Forced Timeout on CAS in PendingDispatch list. The CAS Has Not Been Dispatched since the Service Appears to be Unavailable
	... 4 more

The hang was observed with UIMA-AS 2.4.2 so it could be that the bug was there which was fixed in a later version. Went back through JIRAs and found a few that may have caused this hang in 2.4.2: 

UIMA-4830
UIMA-3572
UIMA-3393

All of the above have been fixed in more recent versions of the UIMA-AS. I will keep this JIRA open for a few days and will close with Wont Fix if there are no complains or comments to justify keeping the JIRA open.

-jerry

> Client hangs with more than 1 time-out
> --------------------------------------
>
>                 Key: UIMA-4210
>                 URL: https://issues.apache.org/jira/browse/UIMA-4210
>             Project: UIMA
>          Issue Type: Bug
>          Components: Async Scaleout
>    Affects Versions: 2.4.2AS
>         Environment: Java 7, Mac OS
>            Reporter: Frank Xu
>              Labels: client, hangs
>   Original Estimate: 504h
>  Remaining Estimate: 504h
>
> The client hangs if the execution has two time-outs. After debugging into the issue, we figure out that the resending mechanism has some bugs in it. Here are the detailed description.
> Please review the necessity for the invocation of sendCAS(). In our system, we don't have to resend the CAS to process again. Please provide a configuration so that we don't have to resend the CAS every time there is a time out.
> Whenever there is the first time-out, BaseUIMAAsynchronousEngine_impl#notifyOnTimout() is invoked and it hangs when it tries to invoke sendCas() at line 2385. I believe the reason is that the sendCAS() is a synchronized method and a potential threading issue causes this thread hang over there. Please be noted that this block is also synchronized.
> Then when there is a second time-out, it will be hanging in the very beginning of the method notifyOnTimeout() and cannot clear the time-out CAS from the CAS list, which hangs the entire client.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)