You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@uima.apache.org by "Jerry Cwiklik (JIRA)" <de...@uima.apache.org> on 2011/02/08 17:50:04 UTC

[jira] Created: (UIMA-2038) UIMA AS process does not terminate reliably

UIMA AS process does not terminate reliably
-------------------------------------------

                 Key: UIMA-2038
                 URL: https://issues.apache.org/jira/browse/UIMA-2038
             Project: UIMA
          Issue Type: Bug
          Components: Async Scaleout
    Affects Versions: 2.3.1AS
            Reporter: Jerry Cwiklik
            Assignee: Jerry Cwiklik


UIMA AS two stop options dont seem to work reliably. Neither 's' and 'q' on the command line force a clean shutdown of the process. Actually, there is also a related problem. Namely, when the shutdown succeeds it appears that the Shared Connection that all Spring listeners use is not closed which leads to an ugly exception on the broker console. This happens every time the service is terminated. Review listeners shutdown code and make sure that when the last listener terminates the connection is stopped before the process exits. Also, make sure that uima threads from custom pools are daemon threads to allow the jvm to collect them on shutdown. NOTE: AMQ version 4.x internal threads are not daemon threads and there is special code in the listener to wait for them to stop before exiting. Newer AMQ version use daemon threads so shutting down AMQ is much more reliable.    

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: [jira] Commented: (UIMA-2038) UIMA AS process does not terminate reliably

Posted by Jaroslaw Cwiklik <ui...@gmail.com>.

Hmm it seems that I need to add spring-aop-3.0.3.RELEASE.jar.

JC

2011/2/21 Jörn Kottmann (JIRA) <de...@uima.apache.org>

>
>    [
> https://issues.apache.org/jira/browse/UIMA-2038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12997439#comment-12997439]
>
> Jörn Kottmann commented on UIMA-2038:
> -------------------------------------
>
> Now I tested it on my test system. I put the new jars of a freshly compiled
> 2.3.1 trunk onto my test system.
>
> When pressing "q" I get the following exception:
>
> 2/21/11 3:57:52 PM - 10:
> org.apache.uima.aae.controller.PrimitiveAnalysisEngineController_impl.stop:
> INFO: Controller: SearchEngine has stopped
> Exception in thread "main" java.lang.reflect.InvocationTargetException
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>        at java.lang.reflect.Method.invoke(Method.java:616)
>        at
> org.apache.uima.bootstrap.UimaBootstrap.main(UimaBootstrap.java:98)
> Caused by: java.lang.NoClassDefFoundError:
> org/springframework/aop/support/AopUtils
>        at
> org.springframework.context.event.GenericApplicationListenerAdapter.supportsEventType(GenericApplicationListenerAdapter.java:57)
>        at
> org.springframework.context.event.AbstractApplicationEventMulticaster.supportsEvent(AbstractApplicationEventMulticaster.java:179)
>        at
> org.springframework.context.event.AbstractApplicationEventMulticaster.getApplicationListeners(AbstractApplicationEventMulticaster.java:140)
>        at
> org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:86)
>        at
> org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:303)
>        at
> org.springframework.context.support.AbstractApplicationContext.stop(AbstractApplicationContext.java:1242)
>        at
> org.apache.uima.adapter.jms.activemq.SpringContainerDeployer.undeploy(SpringContainerDeployer.java:493)
>        at
> org.apache.uima.adapter.jms.service.UIMA_Service.main(UIMA_Service.java:422)
>        ... 5 more
> Caused by: java.lang.ClassNotFoundException:
> org.springframework.aop.support.AopUtils
>        at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
>        at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
>        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
>        at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
>        ... 13 more
>
> It looks like I am just missing some spring jar file, didn't you get this
> error? I checked the distribution produced on the trunk version but could
> not see an additional or updated jar file, but maybe I missed something.
>
> > UIMA AS process does not terminate reliably
> > -------------------------------------------
> >
> >                 Key: UIMA-2038
> >                 URL: https://issues.apache.org/jira/browse/UIMA-2038
> >             Project: UIMA
> >          Issue Type: Bug
> >          Components: Async Scaleout
> >    Affects Versions: 2.3.1AS
> >            Reporter: Jerry Cwiklik
> >            Assignee: Jerry Cwiklik
> >             Fix For: 2.3.1AS
> >
> >
> > UIMA AS two stop options dont seem to work reliably. Neither 's' nor 'q'
> on the command line force a clean shutdown of the process. Actually, there
> is also a related problem. Namely, when the shutdown succeeds it appears
> that the Shared Connection that all Spring listeners use is not closed which
> leads to an ugly exception on the broker console. This happens every time
> the service is terminated. Review listeners shutdown code and make sure that
> when the last listener terminates the connection is stopped before the
> process exits. Also, make sure that uima threads from custom pools are
> daemon threads to allow the jvm to collect them on shutdown. NOTE: AMQ
> version 4.x internal threads are not daemon threads and there is special
> code in the listener to wait for them to stop before exiting. Newer AMQ
> version use daemon threads so shutting down AMQ is much more reliable.
>
> --
> This message is automatically generated by JIRA.
> -
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>
>
>

Re: [jira] Commented: (UIMA-2038) UIMA AS process does not terminate reliably

Posted by Jörn Kottmann <ko...@gmail.com>.

On 3/1/11 5:54 PM, Jaroslaw Cwiklik wrote:
> Not sure I fully understand the scenario. Quiesce and stop waits until all
> CASes are processed and only then it proceeds to stop the service. Perhaps
> the scenario you are describing exposes a bug in the UIMA AS. Can you
> describe the scenario in more detail? Are you sending CASes to the service
> while trying to quit the service? Is the Cas Multiplier producing CASes
> while the service is being stopped?

Yes I am sending CASes to the service which are then processed, through
a bug in one of my AEs I got many exceptions thrown randomly.
I then noticed that stopping with kill pid does not seem to work.
In this AAE is no Cas Multiplier.
I tested it again now and it works correctly.

I noticed that this is only be written to the console:
"Uima AS Service Wrapper Caught Kill Signal - Initiating Quiesce and Stop"

Shouldn't that be logged instead?

Jörn

Re: [jira] Commented: (UIMA-2038) UIMA AS process does not terminate reliably

Posted by Jaroslaw Cwiklik <ui...@gmail.com>.

Not sure I fully understand the scenario. Quiesce and stop waits until all
CASes are processed and only then it proceeds to stop the service. Perhaps
the scenario you are describing exposes a bug in the UIMA AS. Can you
describe the scenario in more detail? Are you sending CASes to the service
while trying to quit the service? Is the Cas Multiplier producing CASes
while the service is being stopped?

JC

On Tue, Mar 1, 2011 at 11:39 AM, Jörn Kottmann <ko...@gmail.com> wrote:

> On 3/1/11 5:29 PM, Jaroslaw Cwiklik wrote:
>
>> Well, I would like to wait a little while longer to give others a chance
>> to
>> comment on RC5. Go ahead with the PID modification. As I recall this is
>> about formatting only. Seems like a tough one :)
>>
>
> I might found one more problem related to this issue with RC5.
>
> I noticed that a simple kill does not work when an AE is constantly
> throwing a RuntimeException
> for many of its processed CASes. As far as I understand the kill triggers
> the shutdown hook which
> does the quiesce. So even when there are exceptions from the process method
> it should be
> possible to quiesce the service.
>
> I will investigate that tomorrow with my now deployed snapshot version.
>
> Jörn
>

Re: [jira] Commented: (UIMA-2038) UIMA AS process does not terminate reliably

Posted by Jörn Kottmann <ko...@gmail.com>.

On 3/1/11 5:29 PM, Jaroslaw Cwiklik wrote:
> Well, I would like to wait a little while longer to give others a chance to
> comment on RC5. Go ahead with the PID modification. As I recall this is
> about formatting only. Seems like a tough one :)

I might found one more problem related to this issue with RC5.

I noticed that a simple kill does not work when an AE is constantly 
throwing a RuntimeException
for many of its processed CASes. As far as I understand the kill 
triggers the shutdown hook which
does the quiesce. So even when there are exceptions from the process 
method it should be
possible to quiesce the service.

I will investigate that tomorrow with my now deployed snapshot version.

Jörn

Re: [jira] Commented: (UIMA-2038) UIMA AS process does not terminate reliably

Posted by Jaroslaw Cwiklik <ui...@gmail.com>.

Well, I would like to wait a little while longer to give others a chance to
comment on RC5. Go ahead with the PID modification. As I recall this is
about formatting only. Seems like a tough one :)

JC

On Tue, Mar 1, 2011 at 11:25 AM, Jörn Kottmann <ko...@gmail.com> wrote:

> On 3/1/11 5:05 PM, Jaroslaw Cwiklik wrote:
>
>> Jorn, I just committed a few changes. These should fix the hang yu've
>> reported. If it works close the JIRA
>> Thanks, JC
>>
>
> Works now, I already closed the issue. Do you make a new RC?
> If so please drop me a note before, would still like to modify the PID
> log message on start up.
>
> Jörn
>

Re: [jira] Commented: (UIMA-2038) UIMA AS process does not terminate reliably

Posted by Jörn Kottmann <ko...@gmail.com>.

On 3/1/11 5:05 PM, Jaroslaw Cwiklik wrote:
> Jorn, I just committed a few changes. These should fix the hang yu've
> reported. If it works close the JIRA
> Thanks, JC

Works now, I already closed the issue. Do you make a new RC?
If so please drop me a note before, would still like to modify the PID
log message on start up.

Jörn