You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@geode.apache.org by "Thacker, Dharam" <dh...@jpmorgan.com> on 2017/02/09 10:42:57 UTC
OutOfMemoryError with cq listener
Hi Team,
I am facing below error while working with continuous query listener. We are receiving high number of events matching to various continuous query listeners which result into below exception. Though task every thread s doing is really very small (Not at all time consuming).
Any guideline on below one? Is it like thread after finishing work not going to pool?
<gfe:cq-listener-container id="cqListenerContainer" cache="event-cache">
<gfe:listener ref="eventListener" query="SELECT * FROM /Event WHERE entityType='A' and entitySubType IN SET ('A1','A2') and status='PENDING'"/>
</gfe:cq-listener-container>
[http-nio-8088-exec-14] ERROR o.a.c.c.C.[.[.[.[dispatcherServlet] Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: unable to create new native thread] with root cause
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
at java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1587)
at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:334)
at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
at org.apache.geode.cache.client.internal.QueueManagerImpl.scheduleRedundancySatisfierIfNeeded(QueueManagerImpl.java:1041)
at org.apache.geode.cache.client.internal.QueueManagerImpl.checkEndpoint(QueueManagerImpl.java:430)
at org.apache.geode.internal.cache.tier.sockets.CacheClientUpdater.processMessages(CacheClientUpdater.java:1811)
at org.apache.geode.internal.cache.tier.sockets.CacheClientUpdater.run(CacheClientUpdater.java:463)
Thanks & Regards,
Dharam
This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email
Re: OutOfMemoryError with cq listener
Posted by Anilkumar Gingade <ag...@pivotal.io>.
What is your client (CQ Listener) doing? Is it storing data or holding
reference to data for long (memory leak?)...Does listener has any logic
that takes long time, reason it may not be catching up with incoming
data...You can also think about handing the event to separate thread
pools....
-Anil.
On Fri, Feb 10, 2017 at 12:40 PM, Barry Oglesby <bo...@pivotal.io> wrote:
> Dan's question is a good one. I was thinking that the issue with the
> CacheClientUpdater dying was happening a lot and causing the client threads
> to spin out of control. But if you're not seeing a lot of messages like
> below in your client log, thats not the case.
>
> [info 2017/02/10 11:07:13.124 GMT <Cache Client Updater Thread on xxx>
> tid=0x22] Primary subscription endpoint xxx crashed. Scheduling recovery.
>
> If your not seeing a lot of message like this, then your client is right
> on the edge of its resources.
>
> The 'java.lang.OutOfMemoryError: unable to create new native thread' means
> the OS doesn't have the resources to allocate a thread. It could be several
> things:
>
> - the physical memory on the box is exhausted
> - the user process limit is reached (ulimit -u)
>
> What is the physical memory on your client machine? What are your JVM's
> -Xms and -Xms settings? Is there enough room between the physical memory
> and the JVM resources for thread stacks? Are you setting -Xss (thread stack
> size)? You could reduce that to save resources. But before you do anything,
> you need to figure out what the threads are.
>
> Setting remove-unresponsive-clients=true will remove a client whose queue
> is full. If a client's queue fills up, puts into it are blocked. Which
> means other client requests are blocked. The main point of setting this
> property is to prevent cascading issues when a client's queue is full at
> the expense of causing that client to reconnect. So, there are pros and
> cons to this property.
>
> The maximum-message-count property limits the maximum size of the client's
> queue before puts block. The default is 230,000 which is mainly to handle
> bursts of activity. A client's queue should never be this full in normal
> processing.
>
> The Cache Client Updater will try forever to recover its connection to the
> server.
>
> The QueueManagerImpl uses a ScheduledThreadPoolExecutor with a maximum
> pool size of Integer.MAX_VALUE, so the number of available threads in this
> area is limited only by the JVM and OS resources.
>
>
> Thanks,
> Barry Oglesby
>
>
> On Fri, Feb 10, 2017 at 6:27 AM, Thacker, Dharam <
> dharam.thacker@jpmorgan.com> wrote:
>
>> Thanks Dan & Barry!
>>
>> Could not reproduce this issue today but got some hints based on your
>> guidance with remove-unresponsive-clients and max-message-count.
>>
>> Somehow logs didn't give useful information, but I will enable client
>> stats and verify it once again.
>>
>> More could you help me to understand max-message-count's role/meaning if
>> I enable remove-unresponsive-clients?
>>
>> More how many times thread will try to recover connection in case it
>> crashes?
>>
>> Is there any way to increase number of threads than what comes by default
>> ? (Even is that recommend?)
>>
>> Thanks,
>> Dharam
>>
>> Sent with Good Work (www.blackberry.com)
>> On Feb 9, 2017 11:50 PM, Dan Smith <ds...@pivotal.io> wrote:
>> The error says "unable to create a new native thread." Do you know how
>> many threads you had in this process when it died? The client stats could
>> show the you the number of threads at least. You might also have an
>> hs_err_pid file that might list the threads.
>>
>> -Dan
>>
>> On Thu, Feb 9, 2017 at 9:23 AM, Barry Oglesby <bo...@pivotal.io>
>> wrote:
>>
>>> Dharam,
>>>
>>> This stack means that the CacheClientUpdater (which is processing the
>>> messages on the client) has caught an exception, is crashing and kicking
>>> off a thread to recover its connection to the server. A high number of
>>> events alone wouldn't cause this unless the client's queue on the server is
>>> full and remove-unresponsive-client=true in the server. Is that the
>>> case? It would be interesting to see the client and server logs as well as
>>> the server stats to see if we can determine why this is happening. The
>>> client log may contain the exception that is causing the CacheClientUpdater
>>> to crash.
>>>
>>> Thanks,
>>> Barry Oglesby
>>>
>>>
>>> On Thu, Feb 9, 2017 at 2:42 AM, Thacker, Dharam <
>>> dharam.thacker@jpmorgan.com> wrote:
>>>
>>>> Hi Team,
>>>>
>>>>
>>>>
>>>> I am facing below error while working with continuous query listener.
>>>> We are receiving high number of events matching to various continuous query
>>>> listeners which result into below exception. Though task every thread s
>>>> doing is really very small (Not at all time consuming).
>>>>
>>>>
>>>>
>>>> Any guideline on below one? Is it like thread after finishing work not
>>>> going to pool?
>>>>
>>>>
>>>>
>>>> <gfe:cq-listener-container id=*"cqListenerContainer"* cache=
>>>> *"event-cache"*>
>>>>
>>>> <gfe:listener ref=*"eventListener"* query=*"SELECT *
>>>> FROM /Event WHERE entityType=’A’ and entitySubType IN SET (‘A1’,’A2') and
>>>> status='PENDING'"*/>
>>>>
>>>> </gfe:cq-listener-container>
>>>>
>>>>
>>>>
>>>> [http-nio-8088-exec-14] ERROR o.a.c.c.C.[.[.[.[dispatcherServlet]
>>>> Servlet.service() for servlet [dispatcherServlet] in context with path []
>>>> threw exception [Handler dispatch failed; nested exception is
>>>> java.lang.OutOfMemoryError: unable to create new native thread] with root
>>>> cause
>>>>
>>>> java.lang.OutOfMemoryError: unable to create new native thread
>>>>
>>>> at java.lang.Thread.start0(Native Method)
>>>>
>>>> at java.lang.Thread.start(Thread.java:714)
>>>>
>>>> at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPool
>>>> Executor.java:950)
>>>>
>>>> at java.util.concurrent.ThreadPoolExecutor.ensurePrestart(Threa
>>>> dPoolExecutor.java:1587)
>>>>
>>>> at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExec
>>>> ute(ScheduledThreadPoolExecutor.java:334)
>>>>
>>>> at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(Sc
>>>> heduledThreadPoolExecutor.java:533)
>>>>
>>>> at org.apache.geode.cache.client.internal.QueueManagerImpl.sche
>>>> duleRedundancySatisfierIfNeeded(QueueManagerImpl.java:1041)
>>>>
>>>> at org.apache.geode.cache.client.internal.QueueManagerImpl.chec
>>>> kEndpoint(QueueManagerImpl.java:430)
>>>>
>>>> at org.apache.geode.internal.cache.tier.sockets.CacheClientUpda
>>>> ter.processMessages(CacheClientUpdater.java:1811)
>>>>
>>>> at org.apache.geode.internal.cache.tier.sockets.CacheClientUpda
>>>> ter.run(CacheClientUpdater.java:463)
>>>>
>>>>
>>>>
>>>> Thanks & Regards,
>>>>
>>>> Dharam
>>>>
>>>> This email is confidential and subject to important disclaimers and
>>>> conditions including on offers for the purchase or sale of securities,
>>>> accuracy and completeness of information, viruses, confidentiality, legal
>>>> privilege, and legal entity disclaimers, available at
>>>> http://www.jpmorgan.com/pages/disclosures/email
>>>>
>>>
>>>
>> This email is confidential and subject to important disclaimers and
>> conditions including on offers for the purchase or sale of securities,
>> accuracy and completeness of information, viruses, confidentiality, legal
>> privilege, and legal entity disclaimers, available at
>> http://www.jpmorgan.com/pages/disclosures/email
>>
>
>
Re: OutOfMemoryError with cq listener
Posted by Barry Oglesby <bo...@pivotal.io>.
Dan's question is a good one. I was thinking that the issue with the
CacheClientUpdater dying was happening a lot and causing the client threads
to spin out of control. But if you're not seeing a lot of messages like
below in your client log, thats not the case.
[info 2017/02/10 11:07:13.124 GMT <Cache Client Updater Thread on xxx>
tid=0x22] Primary subscription endpoint xxx crashed. Scheduling recovery.
If your not seeing a lot of message like this, then your client is right on
the edge of its resources.
The 'java.lang.OutOfMemoryError: unable to create new native thread' means
the OS doesn't have the resources to allocate a thread. It could be several
things:
- the physical memory on the box is exhausted
- the user process limit is reached (ulimit -u)
What is the physical memory on your client machine? What are your JVM's
-Xms and -Xms settings? Is there enough room between the physical memory
and the JVM resources for thread stacks? Are you setting -Xss (thread stack
size)? You could reduce that to save resources. But before you do anything,
you need to figure out what the threads are.
Setting remove-unresponsive-clients=true will remove a client whose queue
is full. If a client's queue fills up, puts into it are blocked. Which
means other client requests are blocked. The main point of setting this
property is to prevent cascading issues when a client's queue is full at
the expense of causing that client to reconnect. So, there are pros and
cons to this property.
The maximum-message-count property limits the maximum size of the client's
queue before puts block. The default is 230,000 which is mainly to handle
bursts of activity. A client's queue should never be this full in normal
processing.
The Cache Client Updater will try forever to recover its connection to the
server.
The QueueManagerImpl uses a ScheduledThreadPoolExecutor with a maximum pool
size of Integer.MAX_VALUE, so the number of available threads in this area
is limited only by the JVM and OS resources.
Thanks,
Barry Oglesby
On Fri, Feb 10, 2017 at 6:27 AM, Thacker, Dharam <
dharam.thacker@jpmorgan.com> wrote:
> Thanks Dan & Barry!
>
> Could not reproduce this issue today but got some hints based on your
> guidance with remove-unresponsive-clients and max-message-count.
>
> Somehow logs didn't give useful information, but I will enable client
> stats and verify it once again.
>
> More could you help me to understand max-message-count's role/meaning if I
> enable remove-unresponsive-clients?
>
> More how many times thread will try to recover connection in case it
> crashes?
>
> Is there any way to increase number of threads than what comes by default
> ? (Even is that recommend?)
>
> Thanks,
> Dharam
>
> Sent with Good Work (www.blackberry.com)
> On Feb 9, 2017 11:50 PM, Dan Smith <ds...@pivotal.io> wrote:
> The error says "unable to create a new native thread." Do you know how
> many threads you had in this process when it died? The client stats could
> show the you the number of threads at least. You might also have an
> hs_err_pid file that might list the threads.
>
> -Dan
>
> On Thu, Feb 9, 2017 at 9:23 AM, Barry Oglesby <bo...@pivotal.io> wrote:
>
>> Dharam,
>>
>> This stack means that the CacheClientUpdater (which is processing the
>> messages on the client) has caught an exception, is crashing and kicking
>> off a thread to recover its connection to the server. A high number of
>> events alone wouldn't cause this unless the client's queue on the server is
>> full and remove-unresponsive-client=true in the server. Is that the
>> case? It would be interesting to see the client and server logs as well as
>> the server stats to see if we can determine why this is happening. The
>> client log may contain the exception that is causing the CacheClientUpdater
>> to crash.
>>
>> Thanks,
>> Barry Oglesby
>>
>>
>> On Thu, Feb 9, 2017 at 2:42 AM, Thacker, Dharam <
>> dharam.thacker@jpmorgan.com> wrote:
>>
>>> Hi Team,
>>>
>>>
>>>
>>> I am facing below error while working with continuous query listener. We
>>> are receiving high number of events matching to various continuous query
>>> listeners which result into below exception. Though task every thread s
>>> doing is really very small (Not at all time consuming).
>>>
>>>
>>>
>>> Any guideline on below one? Is it like thread after finishing work not
>>> going to pool?
>>>
>>>
>>>
>>> <gfe:cq-listener-container id=*"cqListenerContainer"* cache=
>>> *"event-cache"*>
>>>
>>> <gfe:listener ref=*"eventListener"* query=*"SELECT * FROM
>>> /Event WHERE entityType=’A’ and entitySubType IN SET (‘A1’,’A2') and
>>> status='PENDING'"*/>
>>>
>>> </gfe:cq-listener-container>
>>>
>>>
>>>
>>> [http-nio-8088-exec-14] ERROR o.a.c.c.C.[.[.[.[dispatcherServlet]
>>> Servlet.service() for servlet [dispatcherServlet] in context with path []
>>> threw exception [Handler dispatch failed; nested exception is
>>> java.lang.OutOfMemoryError: unable to create new native thread] with root
>>> cause
>>>
>>> java.lang.OutOfMemoryError: unable to create new native thread
>>>
>>> at java.lang.Thread.start0(Native Method)
>>>
>>> at java.lang.Thread.start(Thread.java:714)
>>>
>>> at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPool
>>> Executor.java:950)
>>>
>>> at java.util.concurrent.ThreadPoolExecutor.ensurePrestart(Threa
>>> dPoolExecutor.java:1587)
>>>
>>> at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExec
>>> ute(ScheduledThreadPoolExecutor.java:334)
>>>
>>> at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(Sc
>>> heduledThreadPoolExecutor.java:533)
>>>
>>> at org.apache.geode.cache.client.internal.QueueManagerImpl.sche
>>> duleRedundancySatisfierIfNeeded(QueueManagerImpl.java:1041)
>>>
>>> at org.apache.geode.cache.client.internal.QueueManagerImpl.chec
>>> kEndpoint(QueueManagerImpl.java:430)
>>>
>>> at org.apache.geode.internal.cache.tier.sockets.CacheClientUpda
>>> ter.processMessages(CacheClientUpdater.java:1811)
>>>
>>> at org.apache.geode.internal.cache.tier.sockets.CacheClientUpda
>>> ter.run(CacheClientUpdater.java:463)
>>>
>>>
>>>
>>> Thanks & Regards,
>>>
>>> Dharam
>>>
>>> This email is confidential and subject to important disclaimers and
>>> conditions including on offers for the purchase or sale of securities,
>>> accuracy and completeness of information, viruses, confidentiality, legal
>>> privilege, and legal entity disclaimers, available at
>>> http://www.jpmorgan.com/pages/disclosures/email
>>>
>>
>>
> This email is confidential and subject to important disclaimers and
> conditions including on offers for the purchase or sale of securities,
> accuracy and completeness of information, viruses, confidentiality, legal
> privilege, and legal entity disclaimers, available at
> http://www.jpmorgan.com/pages/disclosures/email
>
Re: OutOfMemoryError with cq listener
Posted by "Thacker, Dharam" <dh...@jpmorgan.com>.
Thanks Dan & Barry!
Could not reproduce this issue today but got some hints based on your guidance with remove-unresponsive-clients and max-message-count.
Somehow logs didn't give useful information, but I will enable client stats and verify it once again.
More could you help me to understand max-message-count's role/meaning if I enable remove-unresponsive-clients?
More how many times thread will try to recover connection in case it crashes?
Is there any way to increase number of threads than what comes by default ? (Even is that recommend?)
Thanks,
Dharam
Sent with Good Work (www.blackberry.com)
On Feb 9, 2017 11:50 PM, Dan Smith <ds...@pivotal.io> wrote:
The error says "unable to create a new native thread." Do you know how many threads you had in this process when it died? The client stats could show the you the number of threads at least. You might also have an hs_err_pid file that might list the threads.
-Dan
On Thu, Feb 9, 2017 at 9:23 AM, Barry Oglesby <bo...@pivotal.io>> wrote:
Dharam,
This stack means that the CacheClientUpdater (which is processing the messages on the client) has caught an exception, is crashing and kicking off a thread to recover its connection to the server. A high number of events alone wouldn't cause this unless the client's queue on the server is full and remove-unresponsive-client=true in the server. Is that the case? It would be interesting to see the client and server logs as well as the server stats to see if we can determine why this is happening. The client log may contain the exception that is causing the CacheClientUpdater to crash.
Thanks,
Barry Oglesby
On Thu, Feb 9, 2017 at 2:42 AM, Thacker, Dharam <dh...@jpmorgan.com>> wrote:
Hi Team,
I am facing below error while working with continuous query listener. We are receiving high number of events matching to various continuous query listeners which result into below exception. Though task every thread s doing is really very small (Not at all time consuming).
Any guideline on below one? Is it like thread after finishing work not going to pool?
<gfe:cq-listener-container id="cqListenerContainer" cache="event-cache">
<gfe:listener ref="eventListener" query="SELECT * FROM /Event WHERE entityType=’A’ and entitySubType IN SET (‘A1’,’A2') and status='PENDING'"/>
</gfe:cq-listener-container>
[http-nio-8088-exec-14] ERROR o.a.c.c.C.[.[.[.[dispatcherServlet] Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: unable to create new native thread] with root cause
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
at java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1587)
at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:334)
at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
at org.apache.geode.cache.client.internal.QueueManagerImpl.scheduleRedundancySatisfierIfNeeded(QueueManagerImpl.java:1041)
at org.apache.geode.cache.client.internal.QueueManagerImpl.checkEndpoint(QueueManagerImpl.java:430)
at org.apache.geode.internal.cache.tier.sockets.CacheClientUpdater.processMessages(CacheClientUpdater.java:1811)
at org.apache.geode.internal.cache.tier.sockets.CacheClientUpdater.run(CacheClientUpdater.java:463)
Thanks & Regards,
Dharam
This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email
This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email
Re: OutOfMemoryError with cq listener
Posted by Dan Smith <ds...@pivotal.io>.
The error says "unable to create a new native thread." Do you know how many
threads you had in this process when it died? The client stats could show
the you the number of threads at least. You might also have an hs_err_pid
file that might list the threads.
-Dan
On Thu, Feb 9, 2017 at 9:23 AM, Barry Oglesby <bo...@pivotal.io> wrote:
> Dharam,
>
> This stack means that the CacheClientUpdater (which is processing the
> messages on the client) has caught an exception, is crashing and kicking
> off a thread to recover its connection to the server. A high number of
> events alone wouldn't cause this unless the client's queue on the server is
> full and remove-unresponsive-client=true in the server. Is that the case?
> It would be interesting to see the client and server logs as well as the
> server stats to see if we can determine why this is happening. The client
> log may contain the exception that is causing the CacheClientUpdater to
> crash.
>
> Thanks,
> Barry Oglesby
>
>
> On Thu, Feb 9, 2017 at 2:42 AM, Thacker, Dharam <
> dharam.thacker@jpmorgan.com> wrote:
>
>> Hi Team,
>>
>>
>>
>> I am facing below error while working with continuous query listener. We
>> are receiving high number of events matching to various continuous query
>> listeners which result into below exception. Though task every thread s
>> doing is really very small (Not at all time consuming).
>>
>>
>>
>> Any guideline on below one? Is it like thread after finishing work not
>> going to pool?
>>
>>
>>
>> <gfe:cq-listener-container id=*"cqListenerContainer"* cache=
>> *"event-cache"*>
>>
>> <gfe:listener ref=*"eventListener"* query=*"SELECT * FROM
>> /Event WHERE entityType=’A’ and entitySubType IN SET (‘A1’,’A2') and
>> status='PENDING'"*/>
>>
>> </gfe:cq-listener-container>
>>
>>
>>
>> [http-nio-8088-exec-14] ERROR o.a.c.c.C.[.[.[.[dispatcherServlet]
>> Servlet.service() for servlet [dispatcherServlet] in context with path []
>> threw exception [Handler dispatch failed; nested exception is
>> java.lang.OutOfMemoryError: unable to create new native thread] with root
>> cause
>>
>> java.lang.OutOfMemoryError: unable to create new native thread
>>
>> at java.lang.Thread.start0(Native Method)
>>
>> at java.lang.Thread.start(Thread.java:714)
>>
>> at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPool
>> Executor.java:950)
>>
>> at java.util.concurrent.ThreadPoolExecutor.ensurePrestart(Threa
>> dPoolExecutor.java:1587)
>>
>> at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExec
>> ute(ScheduledThreadPoolExecutor.java:334)
>>
>> at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(Sc
>> heduledThreadPoolExecutor.java:533)
>>
>> at org.apache.geode.cache.client.internal.QueueManagerImpl.sche
>> duleRedundancySatisfierIfNeeded(QueueManagerImpl.java:1041)
>>
>> at org.apache.geode.cache.client.internal.QueueManagerImpl.chec
>> kEndpoint(QueueManagerImpl.java:430)
>>
>> at org.apache.geode.internal.cache.tier.sockets.CacheClientUpda
>> ter.processMessages(CacheClientUpdater.java:1811)
>>
>> at org.apache.geode.internal.cache.tier.sockets.CacheClientUpda
>> ter.run(CacheClientUpdater.java:463)
>>
>>
>>
>> Thanks & Regards,
>>
>> Dharam
>>
>> This email is confidential and subject to important disclaimers and
>> conditions including on offers for the purchase or sale of securities,
>> accuracy and completeness of information, viruses, confidentiality, legal
>> privilege, and legal entity disclaimers, available at
>> http://www.jpmorgan.com/pages/disclosures/email
>>
>
>
Re: OutOfMemoryError with cq listener
Posted by Barry Oglesby <bo...@pivotal.io>.
Dharam,
This stack means that the CacheClientUpdater (which is processing the
messages on the client) has caught an exception, is crashing and kicking
off a thread to recover its connection to the server. A high number of
events alone wouldn't cause this unless the client's queue on the server is
full and remove-unresponsive-client=true in the server. Is that the case?
It would be interesting to see the client and server logs as well as the
server stats to see if we can determine why this is happening. The client
log may contain the exception that is causing the CacheClientUpdater to
crash.
Thanks,
Barry Oglesby
On Thu, Feb 9, 2017 at 2:42 AM, Thacker, Dharam <dharam.thacker@jpmorgan.com
> wrote:
> Hi Team,
>
>
>
> I am facing below error while working with continuous query listener. We
> are receiving high number of events matching to various continuous query
> listeners which result into below exception. Though task every thread s
> doing is really very small (Not at all time consuming).
>
>
>
> Any guideline on below one? Is it like thread after finishing work not
> going to pool?
>
>
>
> <gfe:cq-listener-container id=*"cqListenerContainer"* cache=
> *"event-cache"*>
>
> <gfe:listener ref=*"eventListener"* query=*"SELECT * FROM
> /Event WHERE entityType=’A’ and entitySubType IN SET (‘A1’,’A2') and
> status='PENDING'"*/>
>
> </gfe:cq-listener-container>
>
>
>
> [http-nio-8088-exec-14] ERROR o.a.c.c.C.[.[.[.[dispatcherServlet]
> Servlet.service() for servlet [dispatcherServlet] in context with path []
> threw exception [Handler dispatch failed; nested exception is
> java.lang.OutOfMemoryError: unable to create new native thread] with root
> cause
>
> java.lang.OutOfMemoryError: unable to create new native thread
>
> at java.lang.Thread.start0(Native Method)
>
> at java.lang.Thread.start(Thread.java:714)
>
> at java.util.concurrent.ThreadPoolExecutor.addWorker(
> ThreadPoolExecutor.java:950)
>
> at java.util.concurrent.ThreadPoolExecutor.ensurePrestart(
> ThreadPoolExecutor.java:1587)
>
> at java.util.concurrent.ScheduledThreadPoolExecutor.
> delayedExecute(ScheduledThreadPoolExecutor.java:334)
>
> at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(
> ScheduledThreadPoolExecutor.java:533)
>
> at org.apache.geode.cache.client.internal.QueueManagerImpl.
> scheduleRedundancySatisfierIfNeeded(QueueManagerImpl.java:1041)
>
> at org.apache.geode.cache.client.internal.QueueManagerImpl.
> checkEndpoint(QueueManagerImpl.java:430)
>
> at org.apache.geode.internal.cache.tier.sockets.
> CacheClientUpdater.processMessages(CacheClientUpdater.java:1811)
>
> at org.apache.geode.internal.cache.tier.sockets.
> CacheClientUpdater.run(CacheClientUpdater.java:463)
>
>
>
> Thanks & Regards,
>
> Dharam
>
> This email is confidential and subject to important disclaimers and
> conditions including on offers for the purchase or sale of securities,
> accuracy and completeness of information, viruses, confidentiality, legal
> privilege, and legal entity disclaimers, available at
> http://www.jpmorgan.com/pages/disclosures/email
>