You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@uima.apache.org by "Jörn Kottmann (JIRA)" <de...@uima.apache.org> on 2011/03/01 10:42:37 UTC
[jira] Commented: (UIMA-2038) UIMA AS process does not terminate
reliably
[ https://issues.apache.org/jira/browse/UIMA-2038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000800#comment-13000800 ]
Jörn Kottmann commented on UIMA-2038:
-------------------------------------
Now re-tested with 2.3.1 RC5 and the issue remains. After processing a few CASes the process does not stop when terminated with "q".
Here are the non-daemon jstack threads:
"DestroyJavaVM" prio=10 tid=0x00007f24ec2b4000 nid=0x370d waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"VmThreadGroup563b24df:12e70c47767:-7ff2_SolrcasAE:Reaper" prio=10 tid=0x00007f24ec2b5000 nid=0x395f in Object.wait() [0x00007f24ea9cf000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007f251e611b00> (a org.apache.uima.aae.spi.transport.vm.VmTransport$1)
at org.apache.uima.aae.spi.transport.vm.VmTransport$1.run(VmTransport.java:150)
- locked <0x00007f251e611b00> (a org.apache.uima.aae.spi.transport.vm.VmTransport$1)
I guess the second thread as a UIMA-AS thread which maybe should be daemon or terminated
in some way.
> UIMA AS process does not terminate reliably
> -------------------------------------------
>
> Key: UIMA-2038
> URL: https://issues.apache.org/jira/browse/UIMA-2038
> Project: UIMA
> Issue Type: Bug
> Components: Async Scaleout
> Affects Versions: 2.3.1AS
> Reporter: Jerry Cwiklik
> Assignee: Jerry Cwiklik
> Fix For: 2.3.1AS
>
>
> UIMA AS two stop options dont seem to work reliably. Neither 's' nor 'q' on the command line force a clean shutdown of the process. Actually, there is also a related problem. Namely, when the shutdown succeeds it appears that the Shared Connection that all Spring listeners use is not closed which leads to an ugly exception on the broker console. This happens every time the service is terminated. Review listeners shutdown code and make sure that when the last listener terminates the connection is stopped before the process exits. Also, make sure that uima threads from custom pools are daemon threads to allow the jvm to collect them on shutdown. NOTE: AMQ version 4.x internal threads are not daemon threads and there is special code in the listener to wait for them to stop before exiting. Newer AMQ version use daemon threads so shutting down AMQ is much more reliable.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] Commented: (UIMA-2038) UIMA AS process does not terminate
reliably
Posted by Jörn Kottmann <ko...@gmail.com>.
On 3/1/11 5:54 PM, Jaroslaw Cwiklik wrote:
> Not sure I fully understand the scenario. Quiesce and stop waits until all
> CASes are processed and only then it proceeds to stop the service. Perhaps
> the scenario you are describing exposes a bug in the UIMA AS. Can you
> describe the scenario in more detail? Are you sending CASes to the service
> while trying to quit the service? Is the Cas Multiplier producing CASes
> while the service is being stopped?
Yes I am sending CASes to the service which are then processed, through
a bug in one of my AEs I got many exceptions thrown randomly.
I then noticed that stopping with kill pid does not seem to work.
In this AAE is no Cas Multiplier.
I tested it again now and it works correctly.
I noticed that this is only be written to the console:
"Uima AS Service Wrapper Caught Kill Signal - Initiating Quiesce and Stop"
Shouldn't that be logged instead?
Jörn
Re: [jira] Commented: (UIMA-2038) UIMA AS process does not terminate reliably
Posted by Jaroslaw Cwiklik <ui...@gmail.com>.
Not sure I fully understand the scenario. Quiesce and stop waits until all
CASes are processed and only then it proceeds to stop the service. Perhaps
the scenario you are describing exposes a bug in the UIMA AS. Can you
describe the scenario in more detail? Are you sending CASes to the service
while trying to quit the service? Is the Cas Multiplier producing CASes
while the service is being stopped?
JC
On Tue, Mar 1, 2011 at 11:39 AM, Jörn Kottmann <ko...@gmail.com> wrote:
> On 3/1/11 5:29 PM, Jaroslaw Cwiklik wrote:
>
>> Well, I would like to wait a little while longer to give others a chance
>> to
>> comment on RC5. Go ahead with the PID modification. As I recall this is
>> about formatting only. Seems like a tough one :)
>>
>
> I might found one more problem related to this issue with RC5.
>
> I noticed that a simple kill does not work when an AE is constantly
> throwing a RuntimeException
> for many of its processed CASes. As far as I understand the kill triggers
> the shutdown hook which
> does the quiesce. So even when there are exceptions from the process method
> it should be
> possible to quiesce the service.
>
> I will investigate that tomorrow with my now deployed snapshot version.
>
> Jörn
>
Re: [jira] Commented: (UIMA-2038) UIMA AS process does not terminate
reliably
Posted by Jörn Kottmann <ko...@gmail.com>.
On 3/1/11 5:29 PM, Jaroslaw Cwiklik wrote:
> Well, I would like to wait a little while longer to give others a chance to
> comment on RC5. Go ahead with the PID modification. As I recall this is
> about formatting only. Seems like a tough one :)
I might found one more problem related to this issue with RC5.
I noticed that a simple kill does not work when an AE is constantly
throwing a RuntimeException
for many of its processed CASes. As far as I understand the kill
triggers the shutdown hook which
does the quiesce. So even when there are exceptions from the process
method it should be
possible to quiesce the service.
I will investigate that tomorrow with my now deployed snapshot version.
Jörn
Re: [jira] Commented: (UIMA-2038) UIMA AS process does not terminate reliably
Posted by Jaroslaw Cwiklik <ui...@gmail.com>.
Well, I would like to wait a little while longer to give others a chance to
comment on RC5. Go ahead with the PID modification. As I recall this is
about formatting only. Seems like a tough one :)
JC
On Tue, Mar 1, 2011 at 11:25 AM, Jörn Kottmann <ko...@gmail.com> wrote:
> On 3/1/11 5:05 PM, Jaroslaw Cwiklik wrote:
>
>> Jorn, I just committed a few changes. These should fix the hang yu've
>> reported. If it works close the JIRA
>> Thanks, JC
>>
>
> Works now, I already closed the issue. Do you make a new RC?
> If so please drop me a note before, would still like to modify the PID
> log message on start up.
>
> Jörn
>
Re: [jira] Commented: (UIMA-2038) UIMA AS process does not terminate
reliably
Posted by Jörn Kottmann <ko...@gmail.com>.
On 3/1/11 5:05 PM, Jaroslaw Cwiklik wrote:
> Jorn, I just committed a few changes. These should fix the hang yu've
> reported. If it works close the JIRA
> Thanks, JC
Works now, I already closed the issue. Do you make a new RC?
If so please drop me a note before, would still like to modify the PID
log message on start up.
Jörn
Re: [jira] Commented: (UIMA-2038) UIMA AS process does not terminate reliably
Posted by Jaroslaw Cwiklik <ui...@gmail.com>.
Jorn, I just committed a few changes. These should fix the hang yu've
reported. If it works close the JIRA
Thanks, JC
2011/3/1 Jörn Kottmann (JIRA) <de...@uima.apache.org>
>
> [
> https://issues.apache.org/jira/browse/UIMA-2038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000800#comment-13000800]
>
> Jörn Kottmann commented on UIMA-2038:
> -------------------------------------
>
> Now re-tested with 2.3.1 RC5 and the issue remains. After processing a few
> CASes the process does not stop when terminated with "q".
>
> Here are the non-daemon jstack threads:
>
>
> "DestroyJavaVM" prio=10 tid=0x00007f24ec2b4000 nid=0x370d waiting on
> condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "VmThreadGroup563b24df:12e70c47767:-7ff2_SolrcasAE:Reaper" prio=10
> tid=0x00007f24ec2b5000 nid=0x395f in Object.wait() [0x00007f24ea9cf000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00007f251e611b00> (a
> org.apache.uima.aae.spi.transport.vm.VmTransport$1)
> at
> org.apache.uima.aae.spi.transport.vm.VmTransport$1.run(VmTransport.java:150)
> - locked <0x00007f251e611b00> (a
> org.apache.uima.aae.spi.transport.vm.VmTransport$1)
>
> I guess the second thread as a UIMA-AS thread which maybe should be daemon
> or terminated
> in some way.
>
>
> > UIMA AS process does not terminate reliably
> > -------------------------------------------
> >
> > Key: UIMA-2038
> > URL: https://issues.apache.org/jira/browse/UIMA-2038
> > Project: UIMA
> > Issue Type: Bug
> > Components: Async Scaleout
> > Affects Versions: 2.3.1AS
> > Reporter: Jerry Cwiklik
> > Assignee: Jerry Cwiklik
> > Fix For: 2.3.1AS
> >
> >
> > UIMA AS two stop options dont seem to work reliably. Neither 's' nor 'q'
> on the command line force a clean shutdown of the process. Actually, there
> is also a related problem. Namely, when the shutdown succeeds it appears
> that the Shared Connection that all Spring listeners use is not closed which
> leads to an ugly exception on the broker console. This happens every time
> the service is terminated. Review listeners shutdown code and make sure that
> when the last listener terminates the connection is stopped before the
> process exits. Also, make sure that uima threads from custom pools are
> daemon threads to allow the jvm to collect them on shutdown. NOTE: AMQ
> version 4.x internal threads are not daemon threads and there is special
> code in the listener to wait for them to stop before exiting. Newer AMQ
> version use daemon threads so shutting down AMQ is much more reliable.
>
> --
> This message is automatically generated by JIRA.
> -
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>
>
>