You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@uima.apache.org by Meghana <me...@germinait.com> on 2011/12/01 06:25:58 UTC
Re: UIMA AS: Lost Connection to Broker
Hi Jerry,
After last night's run, the ActiveMQ broker has consumed barely
50-60MB of RAM (in JConsole), whereas the AS has used all of its
allocated RAM (2GB). If I only restart the AS (not the broker), it
works just fine!
Thanks,
..meghana
On 30 November 2011 20:28, Meghana <me...@germinait.com> wrote:
> Hi Jerry,
>
> Yeah, its the default 256M, will try increasing it. The strange thing
> is, restarting the broker makes no difference; the AS remains
> unresponsive. However, if i also restart the AS, then things flow as
> normal.
>
> Could this be because i'm using a 2.4.0 snapshot build instead of a
> stable release?
>
> Thanks,
>
> ..meghana
>
>
>
> On 30 November 2011 20:08, Jaroslaw Cwiklik <ui...@gmail.com> wrote:
>> Meghana, what is your broker's -Xmx setting? On linux,
>> grep ACTIVEMQ_HOME/bin/activemq for ACTIVEMQ_OPTS_MEMORY. If you are
>> running with a default setting (256M) perhaps you need to increase the
>> memory.
>>
>> Jerry
>>
>> On Mon, Nov 28, 2011 at 11:55 PM, Meghana <me...@germinait.com>wrote:
>>
>>> Hi,
>>>
>>> I have an aggregate AE deployed as an AS with a remote primitive. Both
>>> have their queues at the same broker. Clients send requests to the
>>> aggregate using the sendCAS() method. We use the default 5.4.1
>>> ActiveMQ version and configuration distributed with UIMA AS. The AS's
>>> CAS pool size is 2, and so is the client's.
>>>
>>> After working fine for ~10 hours, the AS listener received 25
>>> exceptions of the following form from its collocated delegates:
>>> 2011-11-29 01:54:55,124 ERROR as.AsListener:
>>> org.apache.uima.aae.error.UimaAsDelegateException: ---->
>>> Controller:/AnalysisAggregator Received Exception on
>>> CAS:61b4719b:133ebbc4eb3:-7fd9 From Delegate:APAnnotator :
>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel : 870
>>>
>>> Then, the AS started returning all CASes with status failed, with the
>>> following logs:
>>> Nov 29, 2011 2:41:25 AM org.apache.uima.aae.delegate.Delegate$1
>>> Delegate.TimerTask.run
>>> WARNING: Timeout While Waiting For Reply From Delegate:q_async_ae
>>> Process CAS Request Timed Out. Configured Reply Window Of 1,200,000.
>>> Cas Reference Id:-3c5bfde0:133ebe967b3:-7fe6
>>> Nov 29, 2011 2:41:25 AM
>>> org.apache.uima.adapter.jms.client.ClientServiceDelegate handleError
>>> WARNING: Process Timeout - Uima AS Client Didn't Receive Process Reply
>>> Within Configured Window Of:1,200,000 millis
>>> Nov 29, 2011 2:41:25 AM
>>> org.apache.uima.adapter.jms.client.BaseUIMAAsynchronousEngineCommon_impl
>>> notifyOnTimout
>>> WARNING: Request To Process Cas Has Timed-out. Service
>>> Queue:q_async_ae. Cas Timed-out on host: 192.168.0.121
>>> Nov 29, 2011 2:41:25 AM
>>> org.apache.uima.adapter.jms.client.ActiveMQMessageSender run
>>> INFO: UIMA AS Client Message Dispatcher Sending GetMeta Ping To the Service
>>> Nov 29, 2011 2:41:25 AM
>>> org.apache.uima.adapter.jms.client.BaseUIMAAsynchronousEngineCommon_impl
>>> sendCAS
>>> INFO: Uima AS Client Sent PING Message To Service: q_async_ae
>>> Nov 29, 2011 2:41:25 AM
>>> org.apache.uima.adapter.jms.client.BaseUIMAAsynchronousEngineCommon_impl
>>> handleException
>>> INFO: Received Exception In Message From:UimaASClient Cas
>>> Identifier:-3c5bfde0:133ebe967b3:-7fe3
>>> Exception:org.apache.uima.jms.error.handler.BrokerConnectionException:
>>> Unable To Deliver CAS:-3c5bfde0:133ebe967b3:-7fe3 To Destination.
>>> Connection To Broker tcp://broker_url:61616 Has Been Lost
>>>
>>> The remote delegate hasn't logged any errors, and the broker "seems"
>>> to be up, but it has a few of these in its log:
>>> 2011-11-29 02:13:33,279 [168.0.121:49414] INFO Transport - Transport
>>> failed: java.net.SocketException: Connection reset
>>> java.net.SocketException: Connection reset
>>> at java.net.SocketInputStream.read(SocketInputStream.java:168)
>>> at
>>> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>>> at
>>> org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:575)
>>> at
>>> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>>> at
>>> org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:560)
>>> at java.io.DataInputStream.readInt(DataInputStream.java:370)
>>> at
>>> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>>> at
>>> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:226)
>>> at
>>> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:218)
>>> at
>>> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:201)
>>> at java.lang.Thread.run(Thread.java:619)
>>>
>>> What could be the problem here? Should i update the ActiveMQ distro?
>>> They have retracted the 5.4.1 release due to the bug at
>>> https://issues.apache.org/jira/browse/AMQ-3491 and recommend using
>>> 5.4.3 instead.
>>>
>>> Thanks a lot,
>>>
>>> ..meghana
>>>
>>
>
Re: UIMA AS: Lost Connection to Broker
Posted by Meghana <me...@germinait.com>.
Hi,
Well, some threads do log: "java.lang.OutOfMemoryError: GC overhead
limit exceeded", but the process remains alive/doesn't crash. And if I
connect using JConsole, the annotator threads are all alive and
waiting for input, etc. Will try to reduce heap space and force OOM.
Thanks,
..meghana
On 1 December 2011 19:55, Jaroslaw Cwiklik <ui...@gmail.com> wrote:
> Did UIMA AS process ran out of memory? Do you have jvm dump file to examine
> the memory leak? If you dont, force the OOM by reducing memory for the UIMA
> AS process and use a tool like heapAnalyzer to find the memory leak:
>
> https://www.ibm.com/developerworks/community/groups/service/html/communityview?communityUuid=4544bafe-c7a2-455f-9d43-eb866ea60091
>
> Jerry
>
> On Thu, Dec 1, 2011 at 12:25 AM, Meghana <me...@germinait.com>wrote:
>
>> Hi Jerry,
>>
>> After last night's run, the ActiveMQ broker has consumed barely
>> 50-60MB of RAM (in JConsole), whereas the AS has used all of its
>> allocated RAM (2GB). If I only restart the AS (not the broker), it
>> works just fine!
>>
>> Thanks,
>>
>> ..meghana
>>
>> On 30 November 2011 20:28, Meghana <me...@germinait.com> wrote:
>> > Hi Jerry,
>> >
>> > Yeah, its the default 256M, will try increasing it. The strange thing
>> > is, restarting the broker makes no difference; the AS remains
>> > unresponsive. However, if i also restart the AS, then things flow as
>> > normal.
>> >
>> > Could this be because i'm using a 2.4.0 snapshot build instead of a
>> > stable release?
>> >
>> > Thanks,
>> >
>> > ..meghana
>> >
>> >
>> >
>> > On 30 November 2011 20:08, Jaroslaw Cwiklik <ui...@gmail.com> wrote:
>> >> Meghana, what is your broker's -Xmx setting? On linux,
>> >> grep ACTIVEMQ_HOME/bin/activemq for ACTIVEMQ_OPTS_MEMORY. If you are
>> >> running with a default setting (256M) perhaps you need to increase the
>> >> memory.
>> >>
>> >> Jerry
>> >>
>> >> On Mon, Nov 28, 2011 at 11:55 PM, Meghana <
>> meghana.marathe@germinait.com>wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> I have an aggregate AE deployed as an AS with a remote primitive. Both
>> >>> have their queues at the same broker. Clients send requests to the
>> >>> aggregate using the sendCAS() method. We use the default 5.4.1
>> >>> ActiveMQ version and configuration distributed with UIMA AS. The AS's
>> >>> CAS pool size is 2, and so is the client's.
>> >>>
>> >>> After working fine for ~10 hours, the AS listener received 25
>> >>> exceptions of the following form from its collocated delegates:
>> >>> 2011-11-29 01:54:55,124 ERROR as.AsListener:
>> >>> org.apache.uima.aae.error.UimaAsDelegateException: ---->
>> >>> Controller:/AnalysisAggregator Received Exception on
>> >>> CAS:61b4719b:133ebbc4eb3:-7fd9 From Delegate:APAnnotator :
>> >>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel : 870
>> >>>
>> >>> Then, the AS started returning all CASes with status failed, with the
>> >>> following logs:
>> >>> Nov 29, 2011 2:41:25 AM org.apache.uima.aae.delegate.Delegate$1
>> >>> Delegate.TimerTask.run
>> >>> WARNING: Timeout While Waiting For Reply From Delegate:q_async_ae
>> >>> Process CAS Request Timed Out. Configured Reply Window Of 1,200,000.
>> >>> Cas Reference Id:-3c5bfde0:133ebe967b3:-7fe6
>> >>> Nov 29, 2011 2:41:25 AM
>> >>> org.apache.uima.adapter.jms.client.ClientServiceDelegate handleError
>> >>> WARNING: Process Timeout - Uima AS Client Didn't Receive Process Reply
>> >>> Within Configured Window Of:1,200,000 millis
>> >>> Nov 29, 2011 2:41:25 AM
>> >>>
>> org.apache.uima.adapter.jms.client.BaseUIMAAsynchronousEngineCommon_impl
>> >>> notifyOnTimout
>> >>> WARNING: Request To Process Cas Has Timed-out. Service
>> >>> Queue:q_async_ae. Cas Timed-out on host: 192.168.0.121
>> >>> Nov 29, 2011 2:41:25 AM
>> >>> org.apache.uima.adapter.jms.client.ActiveMQMessageSender run
>> >>> INFO: UIMA AS Client Message Dispatcher Sending GetMeta Ping To the
>> Service
>> >>> Nov 29, 2011 2:41:25 AM
>> >>>
>> org.apache.uima.adapter.jms.client.BaseUIMAAsynchronousEngineCommon_impl
>> >>> sendCAS
>> >>> INFO: Uima AS Client Sent PING Message To Service: q_async_ae
>> >>> Nov 29, 2011 2:41:25 AM
>> >>>
>> org.apache.uima.adapter.jms.client.BaseUIMAAsynchronousEngineCommon_impl
>> >>> handleException
>> >>> INFO: Received Exception In Message From:UimaASClient Cas
>> >>> Identifier:-3c5bfde0:133ebe967b3:-7fe3
>> >>> Exception:org.apache.uima.jms.error.handler.BrokerConnectionException:
>> >>> Unable To Deliver CAS:-3c5bfde0:133ebe967b3:-7fe3 To Destination.
>> >>> Connection To Broker tcp://broker_url:61616 Has Been Lost
>> >>>
>> >>> The remote delegate hasn't logged any errors, and the broker "seems"
>> >>> to be up, but it has a few of these in its log:
>> >>> 2011-11-29 02:13:33,279 [168.0.121:49414] INFO Transport - Transport
>> >>> failed: java.net.SocketException: Connection reset
>> >>> java.net.SocketException: Connection reset
>> >>> at java.net.SocketInputStream.read(SocketInputStream.java:168)
>> >>> at
>> >>>
>> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>> >>> at
>> >>>
>> org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:575)
>> >>> at
>> >>>
>> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>> >>> at
>> >>>
>> org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:560)
>> >>> at java.io.DataInputStream.readInt(DataInputStream.java:370)
>> >>> at
>> >>>
>> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>> >>> at
>> >>>
>> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:226)
>> >>> at
>> >>>
>> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:218)
>> >>> at
>> >>>
>> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:201)
>> >>> at java.lang.Thread.run(Thread.java:619)
>> >>>
>> >>> What could be the problem here? Should i update the ActiveMQ distro?
>> >>> They have retracted the 5.4.1 release due to the bug at
>> >>> https://issues.apache.org/jira/browse/AMQ-3491 and recommend using
>> >>> 5.4.3 instead.
>> >>>
>> >>> Thanks a lot,
>> >>>
>> >>> ..meghana
>> >>>
>> >>
>> >
>>
Re: UIMA AS: Lost Connection to Broker
Posted by Jaroslaw Cwiklik <ui...@gmail.com>.
Did UIMA AS process ran out of memory? Do you have jvm dump file to examine
the memory leak? If you dont, force the OOM by reducing memory for the UIMA
AS process and use a tool like heapAnalyzer to find the memory leak:
https://www.ibm.com/developerworks/community/groups/service/html/communityview?communityUuid=4544bafe-c7a2-455f-9d43-eb866ea60091
Jerry
On Thu, Dec 1, 2011 at 12:25 AM, Meghana <me...@germinait.com>wrote:
> Hi Jerry,
>
> After last night's run, the ActiveMQ broker has consumed barely
> 50-60MB of RAM (in JConsole), whereas the AS has used all of its
> allocated RAM (2GB). If I only restart the AS (not the broker), it
> works just fine!
>
> Thanks,
>
> ..meghana
>
> On 30 November 2011 20:28, Meghana <me...@germinait.com> wrote:
> > Hi Jerry,
> >
> > Yeah, its the default 256M, will try increasing it. The strange thing
> > is, restarting the broker makes no difference; the AS remains
> > unresponsive. However, if i also restart the AS, then things flow as
> > normal.
> >
> > Could this be because i'm using a 2.4.0 snapshot build instead of a
> > stable release?
> >
> > Thanks,
> >
> > ..meghana
> >
> >
> >
> > On 30 November 2011 20:08, Jaroslaw Cwiklik <ui...@gmail.com> wrote:
> >> Meghana, what is your broker's -Xmx setting? On linux,
> >> grep ACTIVEMQ_HOME/bin/activemq for ACTIVEMQ_OPTS_MEMORY. If you are
> >> running with a default setting (256M) perhaps you need to increase the
> >> memory.
> >>
> >> Jerry
> >>
> >> On Mon, Nov 28, 2011 at 11:55 PM, Meghana <
> meghana.marathe@germinait.com>wrote:
> >>
> >>> Hi,
> >>>
> >>> I have an aggregate AE deployed as an AS with a remote primitive. Both
> >>> have their queues at the same broker. Clients send requests to the
> >>> aggregate using the sendCAS() method. We use the default 5.4.1
> >>> ActiveMQ version and configuration distributed with UIMA AS. The AS's
> >>> CAS pool size is 2, and so is the client's.
> >>>
> >>> After working fine for ~10 hours, the AS listener received 25
> >>> exceptions of the following form from its collocated delegates:
> >>> 2011-11-29 01:54:55,124 ERROR as.AsListener:
> >>> org.apache.uima.aae.error.UimaAsDelegateException: ---->
> >>> Controller:/AnalysisAggregator Received Exception on
> >>> CAS:61b4719b:133ebbc4eb3:-7fd9 From Delegate:APAnnotator :
> >>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel : 870
> >>>
> >>> Then, the AS started returning all CASes with status failed, with the
> >>> following logs:
> >>> Nov 29, 2011 2:41:25 AM org.apache.uima.aae.delegate.Delegate$1
> >>> Delegate.TimerTask.run
> >>> WARNING: Timeout While Waiting For Reply From Delegate:q_async_ae
> >>> Process CAS Request Timed Out. Configured Reply Window Of 1,200,000.
> >>> Cas Reference Id:-3c5bfde0:133ebe967b3:-7fe6
> >>> Nov 29, 2011 2:41:25 AM
> >>> org.apache.uima.adapter.jms.client.ClientServiceDelegate handleError
> >>> WARNING: Process Timeout - Uima AS Client Didn't Receive Process Reply
> >>> Within Configured Window Of:1,200,000 millis
> >>> Nov 29, 2011 2:41:25 AM
> >>>
> org.apache.uima.adapter.jms.client.BaseUIMAAsynchronousEngineCommon_impl
> >>> notifyOnTimout
> >>> WARNING: Request To Process Cas Has Timed-out. Service
> >>> Queue:q_async_ae. Cas Timed-out on host: 192.168.0.121
> >>> Nov 29, 2011 2:41:25 AM
> >>> org.apache.uima.adapter.jms.client.ActiveMQMessageSender run
> >>> INFO: UIMA AS Client Message Dispatcher Sending GetMeta Ping To the
> Service
> >>> Nov 29, 2011 2:41:25 AM
> >>>
> org.apache.uima.adapter.jms.client.BaseUIMAAsynchronousEngineCommon_impl
> >>> sendCAS
> >>> INFO: Uima AS Client Sent PING Message To Service: q_async_ae
> >>> Nov 29, 2011 2:41:25 AM
> >>>
> org.apache.uima.adapter.jms.client.BaseUIMAAsynchronousEngineCommon_impl
> >>> handleException
> >>> INFO: Received Exception In Message From:UimaASClient Cas
> >>> Identifier:-3c5bfde0:133ebe967b3:-7fe3
> >>> Exception:org.apache.uima.jms.error.handler.BrokerConnectionException:
> >>> Unable To Deliver CAS:-3c5bfde0:133ebe967b3:-7fe3 To Destination.
> >>> Connection To Broker tcp://broker_url:61616 Has Been Lost
> >>>
> >>> The remote delegate hasn't logged any errors, and the broker "seems"
> >>> to be up, but it has a few of these in its log:
> >>> 2011-11-29 02:13:33,279 [168.0.121:49414] INFO Transport - Transport
> >>> failed: java.net.SocketException: Connection reset
> >>> java.net.SocketException: Connection reset
> >>> at java.net.SocketInputStream.read(SocketInputStream.java:168)
> >>> at
> >>>
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
> >>> at
> >>>
> org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:575)
> >>> at
> >>>
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
> >>> at
> >>>
> org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:560)
> >>> at java.io.DataInputStream.readInt(DataInputStream.java:370)
> >>> at
> >>>
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
> >>> at
> >>>
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:226)
> >>> at
> >>>
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:218)
> >>> at
> >>>
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:201)
> >>> at java.lang.Thread.run(Thread.java:619)
> >>>
> >>> What could be the problem here? Should i update the ActiveMQ distro?
> >>> They have retracted the 5.4.1 release due to the bug at
> >>> https://issues.apache.org/jira/browse/AMQ-3491 and recommend using
> >>> 5.4.3 instead.
> >>>
> >>> Thanks a lot,
> >>>
> >>> ..meghana
> >>>
> >>
> >
>