You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@uima.apache.org by Charles Proefrock <ch...@hotmail.com> on 2008/10/06 23:59:47 UTC
UimaASProcessCasTimeout exception hangs application
When experimenting with the UIMA-AS examples and how the error handling mechanisms work in terms of timeouts due to AS Aggregates taking too long or going offline, we came across a situation in which the RunRemoteAsyncAE hangs and never returns after receiving and processing a UimaASProcessCasTimeout exception. Ultimately, we simply want the system to recover and try the next CAS or return gracefully without having to call a hard System.exit(1).
Our tests are based on the Deploy_MeetingDetectorTAE_RemoteRoomNumber.xml example. All we did was the following:
(1) added “Thread.sleep(5000);” to RoomNumberAnnotator.java process to simulate a longer process.
(2) Executed: startBroker.bat
(3) Executed: deployAsyncService <DIR>\Deploy_RoomNumberAnnotator.xml
(4) Executed: runRemoteAsyncAE tcp://localhost:61616 MeetingDetectorTaeQueue –c <DIR>\FileSystemCollectionReader.xml -t 4 -i
In summary: when setting and triggering the time outs on the RoomNumber remoteAnalysisEngine, the exception is thrown and caught, each additional CAS is tried, and the system exists gracefully. If the UimaAsynchronousEngine within the runRemoteAsyncAE times out, then the system hangs and never returns.
Is there a trick that we’re missing? Is this expected?
- Charles
_________________________________________________________________
See how Windows Mobile brings your life together—at home, work, or on the go.
http://clk.atdmt.com/MRT/go/msnnkwxp1020093182mrt/direct/01/
Re: UimaASProcessCasTimeout exception hangs application
Posted by Burn Lewis <bu...@gmail.com>.
I meant that if you used:
runRemoteAsyncAE tcp://localhost:61616 RoomNumberAnnotatorQueue –c
<DIR>FileSystemCollectionReader.xml -p 1 -t 6 -i
then CASes would be sent one at a time (-p 1) and the 6 sec timeout would
never be reached since each CAS takes 5 secs to process. But with a larger
CAS pool, say -p 3, the -t value would have to be > 3 x 5, say -p 16, to
ensure that the 3rd CAS does not timeout while waiting for the 1st two to
complete. All this just to avoid errors since as Jerry said there is a bug
in the handling of errors when -i is specified.
Yes, a crash in a service looks just the same as a slow service ... no
result returned in the allotted time. Timeouts are usually used to prevent
the application from hanging when a service dies, so are often set much
larger than the expected processing time to avoid abandoning those CASes
that take longer than the average.
When your application is an asynchronous aggregate with multiple remote
delegates, you can set queue-specific timeouts in the deployment descriptor,
e.g.
<remoteAnalysisEngine key="RoomNumber">
<inputQueue endpoint="RoomNumberAnnotatorQueue"
brokerURL="tcp://localhost:61616"/>
<serializer method="xmi"/>
<asyncAggregateErrorConfiguration>
<getMetadataErrors maxRetries="0" timeout="60000"
errorAction="disable"/>
<processCasErrors maxRetries="0" timeout="300000"
continueOnRetryFailure="true" thresholdCount="1" thresholdWindow="0"
thresholdAction="disable"/>
<collectionProcessCompleteErrors timeout="60000"
additionalErrorAction="terminate"/>
</asyncAggregateErrorConfiguration>
</remoteAnalysisEngine>
See section 3.10 in the uima_async_scaleout document. You can use the CDE
in Eclipse to edit deployment descriptors.
- Burn.
On Tue, Oct 7, 2008 at 4:22 PM, Charles Proefrock <ch...@hotmail.com>wrote:
> I don't have ready access to the test environment, so I'll have to go back
> and play with the -t value. It isn't clear if you are saying that a longer
> timeout "will succeed" in avoiding the hanging problem, or "will succeed" in
> repeating the problem we've observed. Clarification on that would help,
> especially if some min or max timeout avoids the problem.
>
> Yes, the queue name should match the name configured in the deploy
> descriptor. I think we had a more complex example running and then
> simplified it before posting, so the queue name documented here was probably
> just a mistype.
> To be a little more specific, we are testing both the natural timeout when
> a deployed process runs too long, and also the scenario where a deployed
> process crashes/exits with ^C. Both seem to generate the same exception
> when the client timeout occurs, so the nature of the remote failure does not
> seem to be in play. This makes sense since the remote is on the other side
> of a message queue, the client does not see a direct connection dropped on
> the crash. It must also mean that the remote is not connected to the
> anything while it is running, so nothing detects the crash itself.
> Confirmation of this understanding would help as well. Are there any
> options on the message queue (initiation or response queues) that would
> allow for configuring timeouts for specific named queues?
> - Charles
>
>
>
>
> > Date: Tue, 7 Oct 2008 10:10:21 -0400> From: burnlewis@gmail.com> To:
> uima-user@incubator.apache.org> Subject: Re: UimaASProcessCasTimeout
> exception hangs application> > I also see that the -t process timeout value
> has to be > 10 for this to> succeed ... I'd have assumed anything > 5 secs
> would be safe, but the> default value of -p 2 means that the client's
> timeout must be 2x the service> process time when 2 CASes are in play, since
> the 2nd is queued immediately> behind the first. Perhaps the default should
> be -p 1 for simplicity.> > P.S. I assume you meant RoomNumberAnnotatorQueue
> as the 2nd arg in step 4.> > On Tue, Oct 7, 2008 at 9:34 AM, Jaroslaw
> Cwiklik <cw...@us.ibm.com> wrote:> > > Charles, this appears to be a bug
> in the UIMA AS client code. The only work> > around that comes to mind right
> now> > is to run without -i. Without -i the runRemoteAE should terminate on>
> > exception.> >> > We will investigate this problem and post a patch on
> apache.> >> > Thanks> >> > Jerry> >
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++> > Jerry
> Cwiklik> > UIMA Extensions> > IBM T.J. Watson Research Center> > Hawtorne,
> NY, 10532> > Tel: 914-784-7665, T/L: 863-7665> > Email: cwiklik@us.ibm.com>
> >> > [image: Inactive hide details for Charles Proefrock <chas.pro@
> hotmail.com>]Charles> > Proefrock <ch...@hotmail.com>> >> >> >> >
> *Charles Proefrock <ch...@hotmail.com>*> >> > 10/06/2008 05:59 PM> >
> Please respond to> > uima-user@incubator.apache.org> >> >> > To> >> > UIMA
> User <ui...@incubator.apache.org>> > cc> >> >> > Subject> >> >
> UimaASProcessCasTimeout exception hangs application> >> >> >> > When
> experimenting with the UIMA-AS examples and how the error handling> >
> mechanisms work in terms of timeouts due to AS Aggregates taking too long
> or> > going offline, we came across a situation in which the
> RunRemoteAsyncAE> > hangs and never returns after receiving and processing
> a> > UimaASProcessCasTimeout exception. Ultimately, we simply want the
> system to> > recover and try the next CAS or return gracefully without
> having to call a> > hard System.exit(1).> >> > Our tests are based on the
> Deploy_MeetingDetectorTAE_RemoteRoomNumber.xml> > example. All we did was
> the following:> > (1) added "Thread.sleep(5000);" to
> RoomNumberAnnotator.java process to> > simulate a longer process.> > (2)
> Executed: startBroker.bat> > (3) Executed: deployAsyncService
> <DIR>\Deploy_RoomNumberAnnotator.xml> > (4) Executed: runRemoteAsyncAE
> tcp://localhost:61616> > MeetingDetectorTaeQueue –c
> <DIR>\FileSystemCollectionReader.xml -t 4 -i> >> > In summary: when setting
> and triggering the time outs on the RoomNumber> > remoteAnalysisEngine, the
> exception is thrown and caught, each additional> > CAS is tried, and the
> system exists gracefully. If the> > UimaAsynchronousEngine within the
> runRemoteAsyncAE times out, then the> > system hangs and never returns.> >>
> > Is there a trick that we're missing? Is this expected?> >> > - Charles> >>
> >> > _________________________________________________________________> >
> See how Windows Mobile brings your life together—at home, work, or on the> >
> go.> > http://clk.atdmt.com/MRT/go/msnnkwxp1020093182mrt/direct/01/> >
> _________________________________________________________________
> See how Windows Mobile brings your life together—at home, work, or on the
> go.
> http://clk.atdmt.com/MRT/go/msnnkwxp1020093182mrt/direct/01/
>
RE: UimaASProcessCasTimeout exception hangs application
Posted by Charles Proefrock <ch...@hotmail.com>.
I don't have ready access to the test environment, so I'll have to go back and play with the -t value. It isn't clear if you are saying that a longer timeout "will succeed" in avoiding the hanging problem, or "will succeed" in repeating the problem we've observed. Clarification on that would help, especially if some min or max timeout avoids the problem.
Yes, the queue name should match the name configured in the deploy descriptor. I think we had a more complex example running and then simplified it before posting, so the queue name documented here was probably just a mistype.
To be a little more specific, we are testing both the natural timeout when a deployed process runs too long, and also the scenario where a deployed process crashes/exits with ^C. Both seem to generate the same exception when the client timeout occurs, so the nature of the remote failure does not seem to be in play. This makes sense since the remote is on the other side of a message queue, the client does not see a direct connection dropped on the crash. It must also mean that the remote is not connected to the anything while it is running, so nothing detects the crash itself. Confirmation of this understanding would help as well. Are there any options on the message queue (initiation or response queues) that would allow for configuring timeouts for specific named queues?
- Charles
> Date: Tue, 7 Oct 2008 10:10:21 -0400> From: burnlewis@gmail.com> To: uima-user@incubator.apache.org> Subject: Re: UimaASProcessCasTimeout exception hangs application> > I also see that the -t process timeout value has to be > 10 for this to> succeed ... I'd have assumed anything > 5 secs would be safe, but the> default value of -p 2 means that the client's timeout must be 2x the service> process time when 2 CASes are in play, since the 2nd is queued immediately> behind the first. Perhaps the default should be -p 1 for simplicity.> > P.S. I assume you meant RoomNumberAnnotatorQueue as the 2nd arg in step 4.> > On Tue, Oct 7, 2008 at 9:34 AM, Jaroslaw Cwiklik <cw...@us.ibm.com> wrote:> > > Charles, this appears to be a bug in the UIMA AS client code. The only work> > around that comes to mind right now> > is to run without -i. Without -i the runRemoteAE should terminate on> > exception.> >> > We will investigate this problem and post a patch on apache.> >> > Thanks> >> > Jerry> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++> > Jerry Cwiklik> > UIMA Extensions> > IBM T.J. Watson Research Center> > Hawtorne, NY, 10532> > Tel: 914-784-7665, T/L: 863-7665> > Email: cwiklik@us.ibm.com> >> > [image: Inactive hide details for Charles Proefrock <ch...@hotmail.com>]Charles> > Proefrock <ch...@hotmail.com>> >> >> >> > *Charles Proefrock <ch...@hotmail.com>*> >> > 10/06/2008 05:59 PM> > Please respond to> > uima-user@incubator.apache.org> >> >> > To> >> > UIMA User <ui...@incubator.apache.org>> > cc> >> >> > Subject> >> > UimaASProcessCasTimeout exception hangs application> >> >> >> > When experimenting with the UIMA-AS examples and how the error handling> > mechanisms work in terms of timeouts due to AS Aggregates taking too long or> > going offline, we came across a situation in which the RunRemoteAsyncAE> > hangs and never returns after receiving and processing a> > UimaASProcessCasTimeout exception. Ultimately, we simply want the system to> > recover and try the next CAS or return gracefully without having to call a> > hard System.exit(1).> >> > Our tests are based on the Deploy_MeetingDetectorTAE_RemoteRoomNumber.xml> > example. All we did was the following:> > (1) added "Thread.sleep(5000);" to RoomNumberAnnotator.java process to> > simulate a longer process.> > (2) Executed: startBroker.bat> > (3) Executed: deployAsyncService <DIR>\Deploy_RoomNumberAnnotator.xml> > (4) Executed: runRemoteAsyncAE tcp://localhost:61616> > MeetingDetectorTaeQueue –c <DIR>\FileSystemCollectionReader.xml -t 4 -i> >> > In summary: when setting and triggering the time outs on the RoomNumber> > remoteAnalysisEngine, the exception is thrown and caught, each additional> > CAS is tried, and the system exists gracefully. If the> > UimaAsynchronousEngine within the runRemoteAsyncAE times out, then the> > system hangs and never returns.> >> > Is there a trick that we're missing? Is this expected?> >> > - Charles> >> >> > _________________________________________________________________> > See how Windows Mobile brings your life together—at home, work, or on the> > go.> > http://clk.atdmt.com/MRT/go/msnnkwxp1020093182mrt/direct/01/> >
_________________________________________________________________
See how Windows Mobile brings your life together—at home, work, or on the go.
http://clk.atdmt.com/MRT/go/msnnkwxp1020093182mrt/direct/01/
Re: UimaASProcessCasTimeout exception hangs application
Posted by Burn Lewis <bu...@gmail.com>.
I also see that the -t process timeout value has to be > 10 for this to
succeed ... I'd have assumed anything > 5 secs would be safe, but the
default value of -p 2 means that the client's timeout must be 2x the service
process time when 2 CASes are in play, since the 2nd is queued immediately
behind the first. Perhaps the default should be -p 1 for simplicity.
P.S. I assume you meant RoomNumberAnnotatorQueue as the 2nd arg in step 4.
On Tue, Oct 7, 2008 at 9:34 AM, Jaroslaw Cwiklik <cw...@us.ibm.com> wrote:
> Charles, this appears to be a bug in the UIMA AS client code. The only work
> around that comes to mind right now
> is to run without -i. Without -i the runRemoteAE should terminate on
> exception.
>
> We will investigate this problem and post a patch on apache.
>
> Thanks
>
> Jerry
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Jerry Cwiklik
> UIMA Extensions
> IBM T.J. Watson Research Center
> Hawtorne, NY, 10532
> Tel: 914-784-7665, T/L: 863-7665
> Email: cwiklik@us.ibm.com
>
> [image: Inactive hide details for Charles Proefrock <ch...@hotmail.com>]Charles
> Proefrock <ch...@hotmail.com>
>
>
>
> *Charles Proefrock <ch...@hotmail.com>*
>
> 10/06/2008 05:59 PM
> Please respond to
> uima-user@incubator.apache.org
>
>
> To
>
> UIMA User <ui...@incubator.apache.org>
> cc
>
>
> Subject
>
> UimaASProcessCasTimeout exception hangs application
>
>
>
> When experimenting with the UIMA-AS examples and how the error handling
> mechanisms work in terms of timeouts due to AS Aggregates taking too long or
> going offline, we came across a situation in which the RunRemoteAsyncAE
> hangs and never returns after receiving and processing a
> UimaASProcessCasTimeout exception. Ultimately, we simply want the system to
> recover and try the next CAS or return gracefully without having to call a
> hard System.exit(1).
>
> Our tests are based on the Deploy_MeetingDetectorTAE_RemoteRoomNumber.xml
> example. All we did was the following:
> (1) added "Thread.sleep(5000);" to RoomNumberAnnotator.java process to
> simulate a longer process.
> (2) Executed: startBroker.bat
> (3) Executed: deployAsyncService <DIR>\Deploy_RoomNumberAnnotator.xml
> (4) Executed: runRemoteAsyncAE tcp://localhost:61616
> MeetingDetectorTaeQueue –c <DIR>\FileSystemCollectionReader.xml -t 4 -i
>
> In summary: when setting and triggering the time outs on the RoomNumber
> remoteAnalysisEngine, the exception is thrown and caught, each additional
> CAS is tried, and the system exists gracefully. If the
> UimaAsynchronousEngine within the runRemoteAsyncAE times out, then the
> system hangs and never returns.
>
> Is there a trick that we're missing? Is this expected?
>
> - Charles
>
>
> _________________________________________________________________
> See how Windows Mobile brings your life together—at home, work, or on the
> go.
> http://clk.atdmt.com/MRT/go/msnnkwxp1020093182mrt/direct/01/
>
Re: UimaASProcessCasTimeout exception hangs application
Posted by Jaroslaw Cwiklik <cw...@us.ibm.com>.
Charles, this appears to be a bug in the UIMA AS client code. The only work
around that comes to mind right now
is to run without -i. Without -i the runRemoteAE should terminate on
exception.
We will investigate this problem and post a patch on apache.
Thanks
Jerry
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Jerry Cwiklik
UIMA Extensions
IBM T.J. Watson Research Center
Hawtorne, NY, 10532
Tel: 914-784-7665, T/L: 863-7665
Email: cwiklik@us.ibm.com
Charles Proefrock
<chas.pro@hotmail
.com> To
UIMA User
10/06/2008 05:59 <ui...@incubator.apache.org>
PM cc
Subject
Please respond to UimaASProcessCasTimeout exception
uima-user@incubat hangs application
or.apache.org
When experimenting with the UIMA-AS examples and how the error handling
mechanisms work in terms of timeouts due to AS Aggregates taking too long
or going offline, we came across a situation in which the RunRemoteAsyncAE
hangs and never returns after receiving and processing a
UimaASProcessCasTimeout exception. Ultimately, we simply want the system
to recover and try the next CAS or return gracefully without having to call
a hard System.exit(1).
Our tests are based on the Deploy_MeetingDetectorTAE_RemoteRoomNumber.xml
example. All we did was the following:
(1) added “Thread.sleep(5000);” to RoomNumberAnnotator.java process to
simulate a longer process.
(2) Executed: startBroker.bat
(3) Executed: deployAsyncService <DIR>\Deploy_RoomNumberAnnotator.xml
(4) Executed: runRemoteAsyncAE tcp://localhost:61616
MeetingDetectorTaeQueue –c <DIR>\FileSystemCollectionReader.xml -t 4 -i
In summary: when setting and triggering the time outs on the RoomNumber
remoteAnalysisEngine, the exception is thrown and caught, each additional
CAS is tried, and the system exists gracefully. If the
UimaAsynchronousEngine within the runRemoteAsyncAE times out, then the
system hangs and never returns.
Is there a trick that we’re missing? Is this expected?
- Charles
_________________________________________________________________
See how Windows Mobile brings your life together—at home, work, or on the
go.
http://clk.atdmt.com/MRT/go/msnnkwxp1020093182mrt/direct/01/