You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@uima.apache.org by Aiman <an...@columbia.edu> on 2013/03/23 16:01:42 UTC
UIMA-AS: sendAndRecieveCAS hangs with no response
Hello all -
I have been able to successfully deploy my pipeline using UIMA-AS, I
tested it through command line tools and received the output CAS. I was
then able to programmatically call the service using Client API both
asynchronous and synchronously. However, for some reason, after few
runs it stopped working.
Now whenever I call sendAndReceiveCAS it just blocks without receiving
any response, viewing uima.log did not help as I only found some INFO
messages without any errors. sendCAS is more of a hit and miss, it
works sometimes and sometimes it does not. I do not understand what
causing this and I don't know how to troubleshoot it. I've tried to
purge activemq queue, restart it, restart the AS service, changed the
queue name. But none of that helped. When I try to terminate the
service (using s+Enter), it also hangs after showing a few lines of
"....... AnalysisEngineInstancePool.destroy() was called".
I know it's getting stuck somewhere, but my pipeline is very
lightweight and it takes a few milliseconds directly without UIMA-AS. I
would apperciate any help in troubleshooting this.
Thanks!
Re: UIMA-AS: sendAndRecieveCAS hangs with no response
Posted by Spico Florin <sp...@gmail.com>.
Hello!
Thank you very much. It works!
Regards,
Florin
On Fri, Mar 29, 2013 at 6:39 PM, Jaroslaw Cwiklik <ui...@gmail.com> wrote:
> When using a single-threaded client which uses synchronous API
> (sendAndReceive call) you need
> just 1 CAS in the cas pool. The call blocks waiting for a reply. Once the
> method returns the client
> must either call CAS.reset() or CAS.release().
>
> Is it possible that your client may be running code like this:
>
> while( <some condition>) {
> CAS cas = client.getCAS()
> sendAndReceive(cas);
> <do something>
> cas.reset();
> }
>
> If yes, than that would explain the hang. The hang would happen because the
> CAS is never returned
> to the pool and subsequent getCAS() call just hangs. The CAS.reset() just
> resets the CAS and doesnt
> return it to the pool. The fix for this would be this:
>
> CAS cas = client.getCAS();
>
> while( <some condition>) {
> sendAndReceive(cas);
> <do something>
> cas.reset();
> }
>
> - JC
>
> On Thu, Mar 28, 2013 at 8:19 AM, Spico Florin <sp...@gmail.com>
> wrote:
>
> > Hello!
> > I'm using UIMA 2.3.1 and I have a similar problem(client hang out)
> when:
> > 1. Using the CAS pool to get/create the new CASes using
> > UimaAsynchronousEngine.getCas()
> > 2. using UimaAsynchronousEngine.sendAndReceiveCAS
> > 3. The pipeline is having an AAE and two Remote AEs connected to it
> > 4. Pipeline is running correctly
> > By looking
> >
> > It seems to me that the engine is hanging waiting to create in new CAS,
> as
> > in the code:
> > cas = engine.getCAS();
> >
> >
> > Looking with the jconsole, I can see the following:
> >
> > State: WAITING on java.util.concurrent.Semaphore$NonfairSync@5e9de959
> > Total blocked: 3 Total waited: 5
> >
> > Stack trace:
> > sun.misc.Unsafe.park(Native Method)
> > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:941)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1261)
> > java.util.concurrent.Semaphore.acquireUninterruptibly(Semaphore.java:309)
> > *
> >
> >
> org.apache.uima.adapter.jms.client.BaseUIMAAsynchronousEngineCommon_impl$CasQueueEntry.getCas(BaseUIMAAsynchronousEngineCommon_impl.java:656)
> > *
> > The bolded text seems to be the reason of the hanging.
> >
> > Can you please look over it? Also, can you please explain better than the
> > existing documentation, what is the mechanism behind the CAS pool used by
> > the client BaseUIMAAsynchronousEngineCommon in the following terms:
> >
> > 1. what should be the pool size depending on the hw?
> > 2. how CAS lifecycle is managed by the pool? Maning creting a new one,
> > releasing, getting from the pool.
> >
> > I look forward for you answers.
> >
> > Regards,
> > Florin
> >
> >
> > *
> > *Thank yoiu
> >
> > On Mon, Mar 25, 2013 at 3:55 PM, Jaroslaw Cwiklik <ui...@gmail.com>
> > wrote:
> >
> > > Hi, do you have a deeply nested CMs in your pipeline? Is the top level
> > > UIMA-AS service (the one the client is sending CASes to) a CM?
> > >
> > > What you can do to debug this is to use jConsole attached to the
> UIMA-AS
> > > process once you observe a hang. Click on Threads tab
> > > and check each thread to determine where it is stuck. Some threads will
> > be
> > > blocking waiting on queues and thats ok. Check if CM thread
> > > is blocking on getCas() which would mean that CASes are not being
> > released.
> > > Also, check the queue ( via jConsole attached to Broker process).
> > > Does it look like the CASes are stuck in the queue?
> > >
> > > -jc
> > >
> > > On Sat, Mar 23, 2013 at 11:29 AM, Aiman <an...@columbia.edu> wrote:
> > >
> > > > Hi all -
> > > >
> > > > After some debugging, it turned out that this is caused by the Cas
> > > > Multipliers that I have in my pipleline. When I remove them from the
> > > > pipeline it performs very well. I was wondering if there are any
> > > > considerations that I have to make when using cas multipliers with
> > > UIMA-AS
> > > > ? Are there any settings that I have to change to the descriptor?
> > > >
> > > > Thanks
> > > >
> > > >
> > > > On 2013-03-23 15:01:42 +0000, Aiman said:
> > > >
> > > > Hello all -
> > > >>
> > > >> I have been able to successfully deploy my pipeline using UIMA-AS, I
> > > >> tested it through command line tools and received the output CAS. I
> > was
> > > >> then able to programmatically call the service using Client API both
> > > >> asynchronous and synchronously. However, for some reason, after few
> > > runs it
> > > >> stopped working.
> > > >>
> > > >> Now whenever I call sendAndReceiveCAS it just blocks without
> receiving
> > > >> any response, viewing uima.log did not help as I only found some
> INFO
> > > >> messages without any errors. sendCAS is more of a hit and miss, it
> > works
> > > >> sometimes and sometimes it does not. I do not understand what
> causing
> > > this
> > > >> and I don't know how to troubleshoot it. I've tried to purge
> activemq
> > > >> queue, restart it, restart the AS service, changed the queue name.
> But
> > > none
> > > >> of that helped. When I try to terminate the service (using s+Enter),
> > it
> > > >> also hangs after showing a few lines of ".......
> > > AnalysisEngineInstancePool.
> > > >> **destroy() was called".
> > > >>
> > > >> I know it's getting stuck somewhere, but my pipeline is very
> > lightweight
> > > >> and it takes a few milliseconds directly without UIMA-AS. I would
> > > >> apperciate any help in troubleshooting this.
> > > >>
> > > >> Thanks!
> > > >>
> > > >
> > > >
> > > >
> > > >
> > >
> >
>
Re: UIMA-AS: sendAndRecieveCAS hangs with no response
Posted by Jaroslaw Cwiklik <ui...@gmail.com>.
When using a single-threaded client which uses synchronous API
(sendAndReceive call) you need
just 1 CAS in the cas pool. The call blocks waiting for a reply. Once the
method returns the client
must either call CAS.reset() or CAS.release().
Is it possible that your client may be running code like this:
while( <some condition>) {
CAS cas = client.getCAS()
sendAndReceive(cas);
<do something>
cas.reset();
}
If yes, than that would explain the hang. The hang would happen because the
CAS is never returned
to the pool and subsequent getCAS() call just hangs. The CAS.reset() just
resets the CAS and doesnt
return it to the pool. The fix for this would be this:
CAS cas = client.getCAS();
while( <some condition>) {
sendAndReceive(cas);
<do something>
cas.reset();
}
- JC
On Thu, Mar 28, 2013 at 8:19 AM, Spico Florin <sp...@gmail.com> wrote:
> Hello!
> I'm using UIMA 2.3.1 and I have a similar problem(client hang out) when:
> 1. Using the CAS pool to get/create the new CASes using
> UimaAsynchronousEngine.getCas()
> 2. using UimaAsynchronousEngine.sendAndReceiveCAS
> 3. The pipeline is having an AAE and two Remote AEs connected to it
> 4. Pipeline is running correctly
> By looking
>
> It seems to me that the engine is hanging waiting to create in new CAS, as
> in the code:
> cas = engine.getCAS();
>
>
> Looking with the jconsole, I can see the following:
>
> State: WAITING on java.util.concurrent.Semaphore$NonfairSync@5e9de959
> Total blocked: 3 Total waited: 5
>
> Stack trace:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:941)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1261)
> java.util.concurrent.Semaphore.acquireUninterruptibly(Semaphore.java:309)
> *
>
> org.apache.uima.adapter.jms.client.BaseUIMAAsynchronousEngineCommon_impl$CasQueueEntry.getCas(BaseUIMAAsynchronousEngineCommon_impl.java:656)
> *
> The bolded text seems to be the reason of the hanging.
>
> Can you please look over it? Also, can you please explain better than the
> existing documentation, what is the mechanism behind the CAS pool used by
> the client BaseUIMAAsynchronousEngineCommon in the following terms:
>
> 1. what should be the pool size depending on the hw?
> 2. how CAS lifecycle is managed by the pool? Maning creting a new one,
> releasing, getting from the pool.
>
> I look forward for you answers.
>
> Regards,
> Florin
>
>
> *
> *Thank yoiu
>
> On Mon, Mar 25, 2013 at 3:55 PM, Jaroslaw Cwiklik <ui...@gmail.com>
> wrote:
>
> > Hi, do you have a deeply nested CMs in your pipeline? Is the top level
> > UIMA-AS service (the one the client is sending CASes to) a CM?
> >
> > What you can do to debug this is to use jConsole attached to the UIMA-AS
> > process once you observe a hang. Click on Threads tab
> > and check each thread to determine where it is stuck. Some threads will
> be
> > blocking waiting on queues and thats ok. Check if CM thread
> > is blocking on getCas() which would mean that CASes are not being
> released.
> > Also, check the queue ( via jConsole attached to Broker process).
> > Does it look like the CASes are stuck in the queue?
> >
> > -jc
> >
> > On Sat, Mar 23, 2013 at 11:29 AM, Aiman <an...@columbia.edu> wrote:
> >
> > > Hi all -
> > >
> > > After some debugging, it turned out that this is caused by the Cas
> > > Multipliers that I have in my pipleline. When I remove them from the
> > > pipeline it performs very well. I was wondering if there are any
> > > considerations that I have to make when using cas multipliers with
> > UIMA-AS
> > > ? Are there any settings that I have to change to the descriptor?
> > >
> > > Thanks
> > >
> > >
> > > On 2013-03-23 15:01:42 +0000, Aiman said:
> > >
> > > Hello all -
> > >>
> > >> I have been able to successfully deploy my pipeline using UIMA-AS, I
> > >> tested it through command line tools and received the output CAS. I
> was
> > >> then able to programmatically call the service using Client API both
> > >> asynchronous and synchronously. However, for some reason, after few
> > runs it
> > >> stopped working.
> > >>
> > >> Now whenever I call sendAndReceiveCAS it just blocks without receiving
> > >> any response, viewing uima.log did not help as I only found some INFO
> > >> messages without any errors. sendCAS is more of a hit and miss, it
> works
> > >> sometimes and sometimes it does not. I do not understand what causing
> > this
> > >> and I don't know how to troubleshoot it. I've tried to purge activemq
> > >> queue, restart it, restart the AS service, changed the queue name. But
> > none
> > >> of that helped. When I try to terminate the service (using s+Enter),
> it
> > >> also hangs after showing a few lines of ".......
> > AnalysisEngineInstancePool.
> > >> **destroy() was called".
> > >>
> > >> I know it's getting stuck somewhere, but my pipeline is very
> lightweight
> > >> and it takes a few milliseconds directly without UIMA-AS. I would
> > >> apperciate any help in troubleshooting this.
> > >>
> > >> Thanks!
> > >>
> > >
> > >
> > >
> > >
> >
>
Re: UIMA-AS: sendAndRecieveCAS hangs with no response
Posted by Spico Florin <sp...@gmail.com>.
Hello!
I'm using UIMA 2.3.1 and I have a similar problem(client hang out) when:
1. Using the CAS pool to get/create the new CASes using
UimaAsynchronousEngine.getCas()
2. using UimaAsynchronousEngine.sendAndReceiveCAS
3. The pipeline is having an AAE and two Remote AEs connected to it
4. Pipeline is running correctly
By looking
It seems to me that the engine is hanging waiting to create in new CAS, as
in the code:
cas = engine.getCAS();
Looking with the jconsole, I can see the following:
State: WAITING on java.util.concurrent.Semaphore$NonfairSync@5e9de959
Total blocked: 3 Total waited: 5
Stack trace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:941)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1261)
java.util.concurrent.Semaphore.acquireUninterruptibly(Semaphore.java:309)
*
org.apache.uima.adapter.jms.client.BaseUIMAAsynchronousEngineCommon_impl$CasQueueEntry.getCas(BaseUIMAAsynchronousEngineCommon_impl.java:656)
*
The bolded text seems to be the reason of the hanging.
Can you please look over it? Also, can you please explain better than the
existing documentation, what is the mechanism behind the CAS pool used by
the client BaseUIMAAsynchronousEngineCommon in the following terms:
1. what should be the pool size depending on the hw?
2. how CAS lifecycle is managed by the pool? Maning creting a new one,
releasing, getting from the pool.
I look forward for you answers.
Regards,
Florin
*
*Thank yoiu
On Mon, Mar 25, 2013 at 3:55 PM, Jaroslaw Cwiklik <ui...@gmail.com> wrote:
> Hi, do you have a deeply nested CMs in your pipeline? Is the top level
> UIMA-AS service (the one the client is sending CASes to) a CM?
>
> What you can do to debug this is to use jConsole attached to the UIMA-AS
> process once you observe a hang. Click on Threads tab
> and check each thread to determine where it is stuck. Some threads will be
> blocking waiting on queues and thats ok. Check if CM thread
> is blocking on getCas() which would mean that CASes are not being released.
> Also, check the queue ( via jConsole attached to Broker process).
> Does it look like the CASes are stuck in the queue?
>
> -jc
>
> On Sat, Mar 23, 2013 at 11:29 AM, Aiman <an...@columbia.edu> wrote:
>
> > Hi all -
> >
> > After some debugging, it turned out that this is caused by the Cas
> > Multipliers that I have in my pipleline. When I remove them from the
> > pipeline it performs very well. I was wondering if there are any
> > considerations that I have to make when using cas multipliers with
> UIMA-AS
> > ? Are there any settings that I have to change to the descriptor?
> >
> > Thanks
> >
> >
> > On 2013-03-23 15:01:42 +0000, Aiman said:
> >
> > Hello all -
> >>
> >> I have been able to successfully deploy my pipeline using UIMA-AS, I
> >> tested it through command line tools and received the output CAS. I was
> >> then able to programmatically call the service using Client API both
> >> asynchronous and synchronously. However, for some reason, after few
> runs it
> >> stopped working.
> >>
> >> Now whenever I call sendAndReceiveCAS it just blocks without receiving
> >> any response, viewing uima.log did not help as I only found some INFO
> >> messages without any errors. sendCAS is more of a hit and miss, it works
> >> sometimes and sometimes it does not. I do not understand what causing
> this
> >> and I don't know how to troubleshoot it. I've tried to purge activemq
> >> queue, restart it, restart the AS service, changed the queue name. But
> none
> >> of that helped. When I try to terminate the service (using s+Enter), it
> >> also hangs after showing a few lines of ".......
> AnalysisEngineInstancePool.
> >> **destroy() was called".
> >>
> >> I know it's getting stuck somewhere, but my pipeline is very lightweight
> >> and it takes a few milliseconds directly without UIMA-AS. I would
> >> apperciate any help in troubleshooting this.
> >>
> >> Thanks!
> >>
> >
> >
> >
> >
>
Re: UIMA-AS: sendAndRecieveCAS hangs with no response
Posted by Jaroslaw Cwiklik <ui...@gmail.com>.
Hi, do you have a deeply nested CMs in your pipeline? Is the top level
UIMA-AS service (the one the client is sending CASes to) a CM?
What you can do to debug this is to use jConsole attached to the UIMA-AS
process once you observe a hang. Click on Threads tab
and check each thread to determine where it is stuck. Some threads will be
blocking waiting on queues and thats ok. Check if CM thread
is blocking on getCas() which would mean that CASes are not being released.
Also, check the queue ( via jConsole attached to Broker process).
Does it look like the CASes are stuck in the queue?
-jc
On Sat, Mar 23, 2013 at 11:29 AM, Aiman <an...@columbia.edu> wrote:
> Hi all -
>
> After some debugging, it turned out that this is caused by the Cas
> Multipliers that I have in my pipleline. When I remove them from the
> pipeline it performs very well. I was wondering if there are any
> considerations that I have to make when using cas multipliers with UIMA-AS
> ? Are there any settings that I have to change to the descriptor?
>
> Thanks
>
>
> On 2013-03-23 15:01:42 +0000, Aiman said:
>
> Hello all -
>>
>> I have been able to successfully deploy my pipeline using UIMA-AS, I
>> tested it through command line tools and received the output CAS. I was
>> then able to programmatically call the service using Client API both
>> asynchronous and synchronously. However, for some reason, after few runs it
>> stopped working.
>>
>> Now whenever I call sendAndReceiveCAS it just blocks without receiving
>> any response, viewing uima.log did not help as I only found some INFO
>> messages without any errors. sendCAS is more of a hit and miss, it works
>> sometimes and sometimes it does not. I do not understand what causing this
>> and I don't know how to troubleshoot it. I've tried to purge activemq
>> queue, restart it, restart the AS service, changed the queue name. But none
>> of that helped. When I try to terminate the service (using s+Enter), it
>> also hangs after showing a few lines of "....... AnalysisEngineInstancePool.
>> **destroy() was called".
>>
>> I know it's getting stuck somewhere, but my pipeline is very lightweight
>> and it takes a few milliseconds directly without UIMA-AS. I would
>> apperciate any help in troubleshooting this.
>>
>> Thanks!
>>
>
>
>
>
Re: UIMA-AS: sendAndRecieveCAS hangs with no response
Posted by Aiman <an...@columbia.edu>.
Hi all -
After some debugging, it turned out that this is caused by the Cas
Multipliers that I have in my pipleline. When I remove them from the
pipeline it performs very well. I was wondering if there are any
considerations that I have to make when using cas multipliers with
UIMA-AS ? Are there any settings that I have to change to the
descriptor?
Thanks
On 2013-03-23 15:01:42 +0000, Aiman said:
> Hello all -
>
> I have been able to successfully deploy my pipeline using UIMA-AS, I
> tested it through command line tools and received the output CAS. I was
> then able to programmatically call the service using Client API both
> asynchronous and synchronously. However, for some reason, after few
> runs it stopped working.
>
> Now whenever I call sendAndReceiveCAS it just blocks without receiving
> any response, viewing uima.log did not help as I only found some INFO
> messages without any errors. sendCAS is more of a hit and miss, it
> works sometimes and sometimes it does not. I do not understand what
> causing this and I don't know how to troubleshoot it. I've tried to
> purge activemq queue, restart it, restart the AS service, changed the
> queue name. But none of that helped. When I try to terminate the
> service (using s+Enter), it also hangs after showing a few lines of
> "....... AnalysisEngineInstancePool.destroy() was called".
>
> I know it's getting stuck somewhere, but my pipeline is very
> lightweight and it takes a few milliseconds directly without UIMA-AS. I
> would apperciate any help in troubleshooting this.
>
> Thanks!