You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cayenne.apache.org by Andrus Adamchik <an...@objectstyle.org> on 2010/05/01 07:09:28 UTC
Re: Large number of EventManager threads causing OutOfMemoryException
I don't think running things on the same server vs. separate server is
an issue. Hw is your business tier implemented though? Is that an EJB
or a web app? Something in its lifecycle makes Cayenne start multiple
times and this is what you need to figure out.
Andrus
On Apr 30, 2010, at 11:14 AM, Victor Leung wrote:
> Hi Andrus,
>
> Thank you for your prompt and insightful reply!
>
> Looking through the thread dumps as suggested, it definitely seems
> like we
> are starting multiple Cayenne stacks / EventManagers -- on point
> (1), the
> number after the last dash is rather low, and on point (2), the
> address of
> the lock are pretty much all different.
>
> In our current test environment (where the problem is occurring), we
> have
> deployed both the web tier and the business tier as separate EARs on
> the
> same server. As mentioned earlier, communication between the two
> tiers is
> through the use of Cayenne Web Service. Is this a supported
> configuration,
> or do we need to deploy the two EARs on separate servers?
>
> Thanks again,
> Vic
>
>
> On Fri, Apr 30, 2010 at 11:47 AM, Andrus Adamchik <andrus@objectstyle.org
> >wrote:
>
>> Hi Vic,
>>
>> There are two possible explanations - EventManager leaking threads
>> (seems
>> rather unlikely) or you are starting multiple Cayenne stacks (or
>> create
>> multiple EventManagers as a side effect of some other action).
>>
>> A few ways to detect this from a thread dump:
>>
>> 1. Check the names of the cayenne-edt- threads. Specifically check
>> the last
>> part of the name - the number after the last dash. E.g. in
>> "cayenne-edt-16165743-0", this number is "0". If this number is
>> non-repeating and increments indefinitely, that may indicate a
>> leak, however
>> if this number is something rather low (e.g. between 0 and 4) and
>> repeats
>> many times, this means that multiple EventManagers are started
>> somehow.
>>
>> 2. Check the address of the lock of the even thread pool:
>>
>>
>> org.apache.cayenne.event.EventManager
>> $DispatchThread.run(EventManager.java:476)
>>> - locked <0x96d28c40> (a
>>> java.util.Collections$SynchronizedList)
>>>
>>
>> (it is "0x96d28c40" in the example above). If it is the same for all
>> threads, then it is a leak. If it is different for most threads,
>> then you
>> have multiple EM's.
>>
>> Andrus
>>
>>
>> On Apr 30, 2010, at 1:07 AM, Victor Leung wrote:
>>
>> Hi all,
>>>
>>> We are using Cayenne 3.0RC3. Our application has a web tier and a
>>> business
>>> /
>>> data access tier (both deployed on Glassfish v2.1.1). Communication
>>> between
>>> the two tiers is through the use of Cayenne Web Service.
>>>
>>> We have been encountering OutOfMemoryException after a couple of
>>> days of
>>> routine usage. Heap dumps show a large number of daemon threads
>>> associated
>>> with EventManager. As an example, there are some 13,000 entries in
>>> the
>>> heap
>>> dump similar to this:
>>>
>>> "cayenne-edt-16165743-0" daemon prio=10 tid=0x576d3800
>>> nid=0x5b2a in Object.wait() [0x847ad000]
>>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>> at java.lang.Object.wait(Native Method)
>>> at
>>>
>>> org.apache.cayenne.event.EventManager
>>> $DispatchThread.run(EventManager.java:476)
>>> - locked <0x96d28c40> (a
>>> java.util.Collections$SynchronizedList)
>>>
>>> There are no evidence of any deadlocked threads.
>>>
>>> We have since turned on monitoring on the JVM. It appears that the
>>> number
>>> of
>>> daemon threads stays more-or-less constant whilst we are just
>>> reading from
>>> the DB, but will creep up slowly over time whenever we do any sort
>>> of DB
>>> updates.
>>>
>>> Any hints as to how we can get around this problem would be much
>>> appreciated!
>>>
>>> Thanks in advance,
>>> Vic
>>>
>>
>>
Re: Large number of EventManager threads causing OutOfMemoryException
Posted by Victor Leung <vi...@digiflex-solutions.com>.
Hi Andrus,
Thank you -- you've hit the nail right on the head!
In our web tier, we've been starting up a new ClientChannel when creating
each session-specific CayenneContext. I've since changed the code so that
all CayenneContexts are now created using a shared ClientChannel instance
(Singleton pattern).
I've run through a couple of user scenarios, and the server's thread count
seem to have stabilised. We'll try and run through some soak testing in the
next 2 weeks, but I'm pretty confident that we've resolved the problem.
Thank you once again.
Cheers,
Vic
On Sat, May 1, 2010 at 3:12 PM, Andrus Adamchik <an...@objectstyle.org>wrote:
> Or actually, client can be a problem as well. Each ClientChannel that you
> create would start an EventManager. So you may want to do
> ClientChannel.getEventManager().shutdown() before a ClientChannel goes out
> of scope.
>
> Andrus
>
>
>
> On May 1, 2010, at 1:09 PM, Andrus Adamchik wrote:
>
> I don't think running things on the same server vs. separate server is an
>> issue. Hw is your business tier implemented though? Is that an EJB or a web
>> app? Something in its lifecycle makes Cayenne start multiple times and this
>> is what you need to figure out.
>>
>> Andrus
>>
>> On Apr 30, 2010, at 11:14 AM, Victor Leung wrote:
>>
>> Hi Andrus,
>>>
>>> Thank you for your prompt and insightful reply!
>>>
>>> Looking through the thread dumps as suggested, it definitely seems like
>>> we
>>> are starting multiple Cayenne stacks / EventManagers -- on point (1), the
>>> number after the last dash is rather low, and on point (2), the address
>>> of
>>> the lock are pretty much all different.
>>>
>>> In our current test environment (where the problem is occurring), we have
>>> deployed both the web tier and the business tier as separate EARs on the
>>> same server. As mentioned earlier, communication between the two tiers is
>>> through the use of Cayenne Web Service. Is this a supported
>>> configuration,
>>> or do we need to deploy the two EARs on separate servers?
>>>
>>> Thanks again,
>>> Vic
>>>
>>>
>>> On Fri, Apr 30, 2010 at 11:47 AM, Andrus Adamchik <
>>> andrus@objectstyle.org>wrote:
>>>
>>> Hi Vic,
>>>>
>>>> There are two possible explanations - EventManager leaking threads
>>>> (seems
>>>> rather unlikely) or you are starting multiple Cayenne stacks (or create
>>>> multiple EventManagers as a side effect of some other action).
>>>>
>>>> A few ways to detect this from a thread dump:
>>>>
>>>> 1. Check the names of the cayenne-edt- threads. Specifically check the
>>>> last
>>>> part of the name - the number after the last dash. E.g. in
>>>> "cayenne-edt-16165743-0", this number is "0". If this number is
>>>> non-repeating and increments indefinitely, that may indicate a leak,
>>>> however
>>>> if this number is something rather low (e.g. between 0 and 4) and
>>>> repeats
>>>> many times, this means that multiple EventManagers are started somehow.
>>>>
>>>> 2. Check the address of the lock of the even thread pool:
>>>>
>>>>
>>>>
>>>> org.apache.cayenne.event.EventManager$DispatchThread.run(EventManager.java:476)
>>>>
>>>>> - locked <0x96d28c40> (a
>>>>> java.util.Collections$SynchronizedList)
>>>>>
>>>>>
>>>> (it is "0x96d28c40" in the example above). If it is the same for all
>>>> threads, then it is a leak. If it is different for most threads, then
>>>> you
>>>> have multiple EM's.
>>>>
>>>> Andrus
>>>>
>>>>
>>>> On Apr 30, 2010, at 1:07 AM, Victor Leung wrote:
>>>>
>>>> Hi all,
>>>>
>>>>>
>>>>> We are using Cayenne 3.0RC3. Our application has a web tier and a
>>>>> business
>>>>> /
>>>>> data access tier (both deployed on Glassfish v2.1.1). Communication
>>>>> between
>>>>> the two tiers is through the use of Cayenne Web Service.
>>>>>
>>>>> We have been encountering OutOfMemoryException after a couple of days
>>>>> of
>>>>> routine usage. Heap dumps show a large number of daemon threads
>>>>> associated
>>>>> with EventManager. As an example, there are some 13,000 entries in the
>>>>> heap
>>>>> dump similar to this:
>>>>>
>>>>> "cayenne-edt-16165743-0" daemon prio=10 tid=0x576d3800
>>>>> nid=0x5b2a in Object.wait() [0x847ad000]
>>>>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.cayenne.event.EventManager$DispatchThread.run(EventManager.java:476)
>>>>> - locked <0x96d28c40> (a
>>>>> java.util.Collections$SynchronizedList)
>>>>>
>>>>> There are no evidence of any deadlocked threads.
>>>>>
>>>>> We have since turned on monitoring on the JVM. It appears that the
>>>>> number
>>>>> of
>>>>> daemon threads stays more-or-less constant whilst we are just reading
>>>>> from
>>>>> the DB, but will creep up slowly over time whenever we do any sort of
>>>>> DB
>>>>> updates.
>>>>>
>>>>> Any hints as to how we can get around this problem would be much
>>>>> appreciated!
>>>>>
>>>>> Thanks in advance,
>>>>> Vic
>>>>>
>>>>>
>>>>
>>>>
>>
>>
>
Re: Large number of EventManager threads causing OutOfMemoryException
Posted by Andrus Adamchik <an...@objectstyle.org>.
Or actually, client can be a problem as well. Each ClientChannel that
you create would start an EventManager. So you may want to do
ClientChannel.getEventManager().shutdown() before a ClientChannel goes
out of scope.
Andrus
On May 1, 2010, at 1:09 PM, Andrus Adamchik wrote:
> I don't think running things on the same server vs. separate server
> is an issue. Hw is your business tier implemented though? Is that an
> EJB or a web app? Something in its lifecycle makes Cayenne start
> multiple times and this is what you need to figure out.
>
> Andrus
>
> On Apr 30, 2010, at 11:14 AM, Victor Leung wrote:
>
>> Hi Andrus,
>>
>> Thank you for your prompt and insightful reply!
>>
>> Looking through the thread dumps as suggested, it definitely seems
>> like we
>> are starting multiple Cayenne stacks / EventManagers -- on point
>> (1), the
>> number after the last dash is rather low, and on point (2), the
>> address of
>> the lock are pretty much all different.
>>
>> In our current test environment (where the problem is occurring),
>> we have
>> deployed both the web tier and the business tier as separate EARs
>> on the
>> same server. As mentioned earlier, communication between the two
>> tiers is
>> through the use of Cayenne Web Service. Is this a supported
>> configuration,
>> or do we need to deploy the two EARs on separate servers?
>>
>> Thanks again,
>> Vic
>>
>>
>> On Fri, Apr 30, 2010 at 11:47 AM, Andrus Adamchik <andrus@objectstyle.org
>> >wrote:
>>
>>> Hi Vic,
>>>
>>> There are two possible explanations - EventManager leaking threads
>>> (seems
>>> rather unlikely) or you are starting multiple Cayenne stacks (or
>>> create
>>> multiple EventManagers as a side effect of some other action).
>>>
>>> A few ways to detect this from a thread dump:
>>>
>>> 1. Check the names of the cayenne-edt- threads. Specifically check
>>> the last
>>> part of the name - the number after the last dash. E.g. in
>>> "cayenne-edt-16165743-0", this number is "0". If this number is
>>> non-repeating and increments indefinitely, that may indicate a
>>> leak, however
>>> if this number is something rather low (e.g. between 0 and 4) and
>>> repeats
>>> many times, this means that multiple EventManagers are started
>>> somehow.
>>>
>>> 2. Check the address of the lock of the even thread pool:
>>>
>>>
>>> org.apache.cayenne.event.EventManager
>>> $DispatchThread.run(EventManager.java:476)
>>>> - locked <0x96d28c40> (a
>>>> java.util.Collections$SynchronizedList)
>>>>
>>>
>>> (it is "0x96d28c40" in the example above). If it is the same for all
>>> threads, then it is a leak. If it is different for most threads,
>>> then you
>>> have multiple EM's.
>>>
>>> Andrus
>>>
>>>
>>> On Apr 30, 2010, at 1:07 AM, Victor Leung wrote:
>>>
>>> Hi all,
>>>>
>>>> We are using Cayenne 3.0RC3. Our application has a web tier and a
>>>> business
>>>> /
>>>> data access tier (both deployed on Glassfish v2.1.1). Communication
>>>> between
>>>> the two tiers is through the use of Cayenne Web Service.
>>>>
>>>> We have been encountering OutOfMemoryException after a couple of
>>>> days of
>>>> routine usage. Heap dumps show a large number of daemon threads
>>>> associated
>>>> with EventManager. As an example, there are some 13,000 entries
>>>> in the
>>>> heap
>>>> dump similar to this:
>>>>
>>>> "cayenne-edt-16165743-0" daemon prio=10 tid=0x576d3800
>>>> nid=0x5b2a in Object.wait() [0x847ad000]
>>>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> at
>>>>
>>>> org.apache.cayenne.event.EventManager
>>>> $DispatchThread.run(EventManager.java:476)
>>>> - locked <0x96d28c40> (a
>>>> java.util.Collections$SynchronizedList)
>>>>
>>>> There are no evidence of any deadlocked threads.
>>>>
>>>> We have since turned on monitoring on the JVM. It appears that
>>>> the number
>>>> of
>>>> daemon threads stays more-or-less constant whilst we are just
>>>> reading from
>>>> the DB, but will creep up slowly over time whenever we do any
>>>> sort of DB
>>>> updates.
>>>>
>>>> Any hints as to how we can get around this problem would be much
>>>> appreciated!
>>>>
>>>> Thanks in advance,
>>>> Vic
>>>>
>>>
>>>
>
>