You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cayenne.apache.org by Ayhan Kondoz <Ay...@freenet-ag.de> on 2007/03/15 14:45:51 UTC
AW: AW: AW: possible bug / memory leak in DispatchQueue and EventManager?
Hi Andrus,
did you log this as a bug? And do you need anything else from me?
At the moment I added a cronjob to restart the WebService every night which prevents this bug from causing any harm but I really would like to fix it properly and remove the cronjob.
Thanx
Ayhan Kondoz
> -----Ursprüngliche Nachricht-----
> Von: Andrus Adamchik [mailto:andrus@objectstyle.org]
> Gesendet: Mittwoch, 21. Februar 2007 15:20
> An: user@cayenne.apache.org
> Betreff: Re: AW: AW: possible bug / memory leak in DispatchQueue and
> EventManager?
>
> 30 is good. If you had a retained DataContext somewhere you'd get the
> same number as the number of stuck invocations.
>
> 1 million of Invocations is bad.
>
> I guess this needs to be logged as a possible bug to investigate,
> with as many details as possible on the JVM version, etc.
>
> Andrus
>
>
> On Feb 21, 2007, at 4:07 PM, Ayhan Kondoz wrote:
> > Yes seems live there are DataContext instances.
> >
> > http://img73.imageshack.us/img73/3068/memta6.jpg
> >
> > I started the GC a few times before I took this screenshot.
> > As you can see there are still 30 DataContext instances. No idea
> > why thought...
> >
> >
> >> -----Ursprüngliche Nachricht-----
> >> Von: Andrus Adamchik [mailto:andrus@objectstyle.org]
> >> Gesendet: Mittwoch, 21. Februar 2007 14:49
> >> An: user@cayenne.apache.org
> >> Betreff: Re: AW: possible bug / memory leak in DispatchQueue and
> >> EventManager?
> >>
> >> Do you see DataContext instances in the memory profile? I wonder how
> >> many of those do you have, as those are the only Cayenne objects that
> >> have hard reference to the listeners.
> >>
> >> Andrus
> >>
> >>
> >> On Feb 21, 2007, at 3:37 PM, Ayhan Kondoz wrote:
> >>
> >>> Hi,
> >>>
> >>> the JVM has 1 GB of memory and i already tried to run the GC
> >>> manually but it doesn't make a difference. I can start the GC from
> >>> the profiler tool but the number of instances does not change.
> >>> I guess that there are some other references to this objects so
> >>> that the weak ref. does not expire.
> >>>
> >>> And there are no EventManager exception in the log files.
> >>>
> >>>
> >>> ayhan
> >>>
> >>>> -----Ursprüngliche Nachricht-----
> >>>> Von: Andrus Adamchik [mailto:andrus@objectstyle.org]
> >>>> Gesendet: Mittwoch, 21. Februar 2007 14:17
> >>>> An: user@cayenne.apache.org
> >>>> Betreff: Re: possible bug / memory leak in DispatchQueue and
> >>>> EventManager?
> >>>>
> >>>> Seems like your assessment of the EventManager leaking is correct.
> >>>>
> >>>> Now the cause is not that clear. A shot in the dark - this is
> >>>> due to
> >>>> a combination of lots of spare memory in JVM (so weak references
> >>>> are
> >>>> not collected fast enough) and slow custom 'equals' and 'hashCode'
> >>>> methods in invocation.
> >>>>
> >>>> How much heap size do you have in your server? Also can you try to
> >>>> add a request filter that would do 'System.gc()' on every few
> >>>> thousands requests, and see if it makes any difference.
> >>>>
> >>>> Also - do you see any EventManager exceptions in the logs?
> >>>>
> >>>> Andrus
> >>>>
> >>>>
> >>>> On Feb 21, 2007, at 10:55 AM, Ayhan Kondoz wrote:
> >>>>> Hi,
> >>>>>
> >>>>>
> >>>>>
> >>>>> i think i found a possible bug / memory leak within the
> >>>>> DispatchQueue and EventManager.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> First a little bit about my setup:
> >>>>>
> >>>>>
> >>>>>
> >>>>> I have 3 servers. Each server runs an axis service. The service
> >>>>> uses cayenne 1.2.1 to connect to a database. It reads customer and
> >>>>> account information from the DB etc..
> >>>>>
> >>>>> The servers are using cayenne's shared caching with javagroups as
> >>>>> the messaging service so that changes made from one server are
> >>>>> dispatched to the other servers.
> >>>>>
> >>>>>
> >>>>>
> >>>>> The avarage connections per second is somewhere around 4-5.
> >>>>>
> >>>>>
> >>>>>
> >>>>> However I have a very strange problem with this setup so I startet
> >>>>> to search for the reason. The problem is that with nearly constant
> >>>>> and unchanging usage the load of each server increases over time.
> >>>>> To further test this I created a test server with a similar setup.
> >>>>> There I created a test program that creates totally constant
> >>>>> usage.
> >>>>> But even with the unchanging usage the load of the server is
> >>>>> increasing until the cpu load is so high that the requests can not
> >>>>> be processed anymore.
> >>>>>
> >>>>>
> >>>>>
> >>>>> I installed a java profiler to trying to pinpoint the location of
> >>>>> this error and this is what I found out.
> >>>>>
> >>>>>
> >>>>>
> >>>>> I let the server run for 24 hours and then stopped the program
> >>>>> which creates the test usage.
> >>>>>
> >>>>> But even while the server was idle there where still a lot of
> >>>>> instances in the java heap after the GC run.
> >>>>>
> >>>>>
> >>>>>
> >>>>> http://img206.imageshack.us/img206/9769/memorysy5.jpg
> >>>>>
> >>>>>
> >>>>>
> >>>>> Please note the HashMap, WeakReference and Invocation counts. I
> >>>>> pressume that the $ObjectProvider_*** is cayenne aswell but I am
> >>>>> not sure.
> >>>>>
> >>>>>
> >>>>>
> >>>>> Now the following image shows the cpu profile with incoming
> >>>>> connections.
> >>>>>
> >>>>>
> >>>>>
> >>>>> http://img339.imageshack.us/img339/2441/cpuci0.jpg
> >>>>>
> >>>>>
> >>>>>
> >>>>> As you can see 58% of the cpu time is used within HashSet.add().
> >>>>>
> >>>>>
> >>>>>
> >>>>> So when I consider the two facts i think that there might be a
> >>>>> possible problem with the EventManager. The first table tells us
> >>>>> that there are over 1 million instances of HashSet's and cayenne
> >>>>> Invocations. So it seems like the set's within the DispatchQueue
> >>>>> are not recycled properly so that the object count rises over time
> >>>>> which result in extremly high processtime when trying to add new
> >>>>> HashSet's.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> Thanks
> >>>>>
> >>>>> Ayhan
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> Ayhan Kondoz
> >>>>>
> >>>>>
> >>>>>
> >>>>> Software-Entwicklung
> >>>>>
> >>>>>
> >>>>>
> >>>>> ------------------------------------------------------------------
> >>>>> --
> >>>>> --
> >>>>> ------------
> >>>>>
> >>>>> Telefon: +49 (0) 40 513 06 616
> >>>>>
> >>>>> Telefax: +49 (0) 40 513 06 998 616
> >>>>>
> >>>>> E-Mail: ayhan.kondoz@freenet-ag.de
> >>>>> <mailto:ayhan.kondoz@freenet-
> >>>>> ag.de>
> >>>>>
> >>>>> Website: http://www.freenet.de <http://www.freenet.de/> ; http://
> >>>>> www.freenet-ag.de <http://www.freenet-ag.de/>
> >>>>>
> >>>>> ------------------------------------------------------------------
> >>>>> --
> >>>>> --
> >>>>> ------------
> >>>>>
> >>>>> freenet.de AG
> >>>>>
> >>>>> Deelbögenkamp 4c
> >>>>>
> >>>>> 22297 Hamburg
> >>>>>
> >>>>> ------------------------------------------------------------------
> >>>>> --
> >>>>> --
> >>>>> ------------
> >>>>>
> >>>>> Vorsitzender des Aufsichtsrates: Prof. Dr. Helmut Thoma
> >>>>>
> >>>>> Vorstand: Eckhard Spoerr (Vors.),
> >>>>> Axel Krieger, Stephan Esch, Eric Berger
> >>>>>
> >>>>> Amtsgericht Hamburg HRB 74048
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> Diese Information ist ausschließlich für die adressierte Person
> >>>>> oder Organisation bestimmt und könnte vertrauliches und/oder
> >>>>> privilegiertes Material enthalten. Personen oder Organisationen,
> >>>>> für die diese Information nicht bestimmt ist, ist es nicht
> >>>>> gestattet, diese zu lesen, erneut zu übertragen, zu verbreiten,
> >>>>> anderweitig zu verwenden oder sich durch sie veranlasst zu sehen,
> >>>>> Maßnahmen irgendeiner Art zu ergreifen. Sollten Sie diese
> >>>>> Nachricht
> >>>>> irrtümlich erhalten haben, bitten wir Sie, sich mit dem
> >>>>> Absender in
> >>>>> Verbindung zu setzen und das Material von Ihrem Computer zu
> >>>>> löschen.
> >>>>>
> >>>>>
> >>>>>
> >>>>> The information transmitted is intended only for the person or
> >>>>> entity to which it is addressed and may contain confidential
> >>>>> and/or
> >>>>> privileged material. Any review, retransmission, dissemination or
> >>>>> other use of, or taking of any action in reliance upon, this
> >>>>> information by persons or entities other than the intended
> >>>>> recipient is prohibited. If you received this in error, please
> >>>>> contact the sender and delete the material from any computer.
> >>>>>
> >>>>>
> >>>>>
> >>>
> >>>
> >
> >
Re: AW: AW: AW: possible bug / memory leak in DispatchQueue and EventManager?
Posted by Andrus Adamchik <an...@objectstyle.org>.
I didn't. I'd appreciate if you could do that, as you have all the
details.
http://issues.apache.org/cayenne/
Thanks
Andrus
On Mar 15, 2007, at 3:45 PM, Ayhan Kondoz wrote:
> Hi Andrus,
>
> did you log this as a bug? And do you need anything else from me?
>
> At the moment I added a cronjob to restart the WebService every
> night which prevents this bug from causing any harm but I really
> would like to fix it properly and remove the cronjob.
>
> Thanx
> Ayhan Kondoz
>
>> -----Ursprüngliche Nachricht-----
>> Von: Andrus Adamchik [mailto:andrus@objectstyle.org]
>> Gesendet: Mittwoch, 21. Februar 2007 15:20
>> An: user@cayenne.apache.org
>> Betreff: Re: AW: AW: possible bug / memory leak in DispatchQueue and
>> EventManager?
>>
>> 30 is good. If you had a retained DataContext somewhere you'd get the
>> same number as the number of stuck invocations.
>>
>> 1 million of Invocations is bad.
>>
>> I guess this needs to be logged as a possible bug to investigate,
>> with as many details as possible on the JVM version, etc.
>>
>> Andrus
>>
>>
>> On Feb 21, 2007, at 4:07 PM, Ayhan Kondoz wrote:
>>> Yes seems live there are DataContext instances.
>>>
>>> http://img73.imageshack.us/img73/3068/memta6.jpg
>>>
>>> I started the GC a few times before I took this screenshot.
>>> As you can see there are still 30 DataContext instances. No idea
>>> why thought...
>>>
>>>
>>>> -----Ursprüngliche Nachricht-----
>>>> Von: Andrus Adamchik [mailto:andrus@objectstyle.org]
>>>> Gesendet: Mittwoch, 21. Februar 2007 14:49
>>>> An: user@cayenne.apache.org
>>>> Betreff: Re: AW: possible bug / memory leak in DispatchQueue and
>>>> EventManager?
>>>>
>>>> Do you see DataContext instances in the memory profile? I wonder
>>>> how
>>>> many of those do you have, as those are the only Cayenne objects
>>>> that
>>>> have hard reference to the listeners.
>>>>
>>>> Andrus
>>>>
>>>>
>>>> On Feb 21, 2007, at 3:37 PM, Ayhan Kondoz wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> the JVM has 1 GB of memory and i already tried to run the GC
>>>>> manually but it doesn't make a difference. I can start the GC from
>>>>> the profiler tool but the number of instances does not change.
>>>>> I guess that there are some other references to this objects so
>>>>> that the weak ref. does not expire.
>>>>>
>>>>> And there are no EventManager exception in the log files.
>>>>>
>>>>>
>>>>> ayhan
>>>>>
>>>>>> -----Ursprüngliche Nachricht-----
>>>>>> Von: Andrus Adamchik [mailto:andrus@objectstyle.org]
>>>>>> Gesendet: Mittwoch, 21. Februar 2007 14:17
>>>>>> An: user@cayenne.apache.org
>>>>>> Betreff: Re: possible bug / memory leak in DispatchQueue and
>>>>>> EventManager?
>>>>>>
>>>>>> Seems like your assessment of the EventManager leaking is
>>>>>> correct.
>>>>>>
>>>>>> Now the cause is not that clear. A shot in the dark - this is
>>>>>> due to
>>>>>> a combination of lots of spare memory in JVM (so weak references
>>>>>> are
>>>>>> not collected fast enough) and slow custom 'equals' and
>>>>>> 'hashCode'
>>>>>> methods in invocation.
>>>>>>
>>>>>> How much heap size do you have in your server? Also can you
>>>>>> try to
>>>>>> add a request filter that would do 'System.gc()' on every few
>>>>>> thousands requests, and see if it makes any difference.
>>>>>>
>>>>>> Also - do you see any EventManager exceptions in the logs?
>>>>>>
>>>>>> Andrus
>>>>>>
>>>>>>
>>>>>> On Feb 21, 2007, at 10:55 AM, Ayhan Kondoz wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> i think i found a possible bug / memory leak within the
>>>>>>> DispatchQueue and EventManager.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> First a little bit about my setup:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I have 3 servers. Each server runs an axis service. The service
>>>>>>> uses cayenne 1.2.1 to connect to a database. It reads
>>>>>>> customer and
>>>>>>> account information from the DB etc..
>>>>>>>
>>>>>>> The servers are using cayenne's shared caching with
>>>>>>> javagroups as
>>>>>>> the messaging service so that changes made from one server are
>>>>>>> dispatched to the other servers.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> The avarage connections per second is somewhere around 4-5.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> However I have a very strange problem with this setup so I
>>>>>>> startet
>>>>>>> to search for the reason. The problem is that with nearly
>>>>>>> constant
>>>>>>> and unchanging usage the load of each server increases over
>>>>>>> time.
>>>>>>> To further test this I created a test server with a similar
>>>>>>> setup.
>>>>>>> There I created a test program that creates totally constant
>>>>>>> usage.
>>>>>>> But even with the unchanging usage the load of the server is
>>>>>>> increasing until the cpu load is so high that the requests
>>>>>>> can not
>>>>>>> be processed anymore.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I installed a java profiler to trying to pinpoint the
>>>>>>> location of
>>>>>>> this error and this is what I found out.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I let the server run for 24 hours and then stopped the program
>>>>>>> which creates the test usage.
>>>>>>>
>>>>>>> But even while the server was idle there where still a lot of
>>>>>>> instances in the java heap after the GC run.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> http://img206.imageshack.us/img206/9769/memorysy5.jpg
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Please note the HashMap, WeakReference and Invocation counts. I
>>>>>>> pressume that the $ObjectProvider_*** is cayenne aswell but I am
>>>>>>> not sure.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Now the following image shows the cpu profile with incoming
>>>>>>> connections.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> http://img339.imageshack.us/img339/2441/cpuci0.jpg
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> As you can see 58% of the cpu time is used within HashSet.add().
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> So when I consider the two facts i think that there might be a
>>>>>>> possible problem with the EventManager. The first table tells us
>>>>>>> that there are over 1 million instances of HashSet's and cayenne
>>>>>>> Invocations. So it seems like the set's within the DispatchQueue
>>>>>>> are not recycled properly so that the object count rises over
>>>>>>> time
>>>>>>> which result in extremly high processtime when trying to add new
>>>>>>> HashSet's.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> Ayhan
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Ayhan Kondoz
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Software-Entwicklung
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ----------------------------------------------------------------
>>>>>>> --
>>>>>>> --
>>>>>>> --
>>>>>>> ------------
>>>>>>>
>>>>>>> Telefon: +49 (0) 40 513 06 616
>>>>>>>
>>>>>>> Telefax: +49 (0) 40 513 06 998 616
>>>>>>>
>>>>>>> E-Mail: ayhan.kondoz@freenet-ag.de
>>>>>>> <mailto:ayhan.kondoz@freenet-
>>>>>>> ag.de>
>>>>>>>
>>>>>>> Website: http://www.freenet.de <http://www.freenet.de/> ;
>>>>>>> http://
>>>>>>> www.freenet-ag.de <http://www.freenet-ag.de/>
>>>>>>>
>>>>>>> ----------------------------------------------------------------
>>>>>>> --
>>>>>>> --
>>>>>>> --
>>>>>>> ------------
>>>>>>>
>>>>>>> freenet.de AG
>>>>>>>
>>>>>>> Deelbögenkamp 4c
>>>>>>>
>>>>>>> 22297 Hamburg
>>>>>>>
>>>>>>> ----------------------------------------------------------------
>>>>>>> --
>>>>>>> --
>>>>>>> --
>>>>>>> ------------
>>>>>>>
>>>>>>> Vorsitzender des Aufsichtsrates: Prof. Dr. Helmut Thoma
>>>>>>>
>>>>>>> Vorstand: Eckhard Spoerr (Vors.),
>>>>>>> Axel Krieger, Stephan Esch, Eric Berger
>>>>>>>
>>>>>>> Amtsgericht Hamburg HRB 74048
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Diese Information ist ausschließlich für die adressierte Person
>>>>>>> oder Organisation bestimmt und könnte vertrauliches und/oder
>>>>>>> privilegiertes Material enthalten. Personen oder Organisationen,
>>>>>>> für die diese Information nicht bestimmt ist, ist es nicht
>>>>>>> gestattet, diese zu lesen, erneut zu übertragen, zu verbreiten,
>>>>>>> anderweitig zu verwenden oder sich durch sie veranlasst zu
>>>>>>> sehen,
>>>>>>> Maßnahmen irgendeiner Art zu ergreifen. Sollten Sie diese
>>>>>>> Nachricht
>>>>>>> irrtümlich erhalten haben, bitten wir Sie, sich mit dem
>>>>>>> Absender in
>>>>>>> Verbindung zu setzen und das Material von Ihrem Computer zu
>>>>>>> löschen.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> The information transmitted is intended only for the person or
>>>>>>> entity to which it is addressed and may contain confidential
>>>>>>> and/or
>>>>>>> privileged material. Any review, retransmission,
>>>>>>> dissemination or
>>>>>>> other use of, or taking of any action in reliance upon, this
>>>>>>> information by persons or entities other than the intended
>>>>>>> recipient is prohibited. If you received this in error, please
>>>>>>> contact the sender and delete the material from any computer.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>
>>>
>
>