You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cayenne.apache.org by Joe Baldwin <jf...@earthlink.net> on 2012/03/13 19:59:21 UTC
Memory Usage Problem with Tomcat
OK, I think that I may have run into this before. The ultimate "solution" was to increase memory - however, I am concerned that may have been a quick-fix and not a long-term fix.
The problem is out of memory errors associated with tomcat heap.
I have a webapp (powered primarily by cayenne). The database has *very* little in it. I am essentially serving data (via cayenne & tomcat) and images (via tomcat).
I have a private tomcat instance running on a webhost in a "shared" environment. What this means is that I *absolutely* cannot attach a profiler.
I am being told by the webhost IT people (who are not always accurate in their objectivity) that my app is leaking memory (badly), and that is what caused tomcat to crash.
My intuition tells me that with almost no activity on the website (because it is not live yet) and Cayenne memory management that I should be able to manage memory well, but it is not the case.
So, if my goal is to determine what the problem is, and if I simply increase heap size, won't I just be masking a potential problem? i.e. if the app runs fine for a while, then mysteriously causes tomcat to run out of heap space, then couldn't there be a memory leak?
If there is a memory leak, and I don't see it on my development server, and I *can't* use a profiler on my webhost, then how do I get visibility into the memory usage?
Thanks
Joe
Re: Memory Usage Problem with Tomcat
Posted by Andrew Lindesay <ap...@lindesay.co.nz>.
Hello;
If CAY-1668 is your issue then it is actually fairly quick to observe
because the memory consumption pattern is exponential with respect to
the quantity of times the ObjectContact is serialized and de-serialized.
The behavior is not actually relative to the amount of work done with
Cayenne itself. Can you ask the service provider if they are always
serializing and de-serializing the sessions behind the scenes in the
shared container?
cheers.
On 14/03/12 12:50 PM, Mike Kienenberger wrote:
> Unfortunately, like Andrew said, it may not be the data objects that
> are leaking. CAY-1668
>
>
> On Tue, Mar 13, 2012 at 7:36 PM, Joe Baldwin<jf...@earthlink.net> wrote:
>> Thanks.
>>
>> Yes, it is hard to tell where to start investigating. I had a theory that if I could log when the Cayenne data objects were being created and destroyed, then I could definitively say whether this is a Cayenne related issue or not.
>>
>> Joe
>>
>>
>> On Mar 13, 2012, at 6:41 PM, Aristedes Maniatis wrote:
>>
>>> On 14/03/12 5:59 AM, Joe Baldwin wrote:
>>>> OK, I think that I may have run into this before. The ultimate "solution" was to increase memory - however, I am concerned that may have been a quick-fix and not a long-term fix.
>>>>
>>>> The problem is out of memory errors associated with tomcat heap.
>>>
>>> Under FreeBSD I had odd memory issues with tomcat 7 until I switched from using the Diablo (Sun approved) JDK and instead used the openjdk6. Then all the memory leak issues went away.
>>>
>>> Mind you I am still stuck with other issues to do with parallel deployment in tomcat7 but I don't think they have anything to do with Cayenne.
>>>
>>> Ari
>>>
>>>
>>>
>>>
>>> --
>>> -------------------------->
>>> Aristedes Maniatis
>>> GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
>>
>
--
Andrew Lindesay
www.silvereye.co.nz
Re: Memory Usage Problem with Tomcat
Posted by Mike Kienenberger <mk...@gmail.com>.
Unfortunately, like Andrew said, it may not be the data objects that
are leaking. CAY-1668
On Tue, Mar 13, 2012 at 7:36 PM, Joe Baldwin <jf...@earthlink.net> wrote:
> Thanks.
>
> Yes, it is hard to tell where to start investigating. I had a theory that if I could log when the Cayenne data objects were being created and destroyed, then I could definitively say whether this is a Cayenne related issue or not.
>
> Joe
>
>
> On Mar 13, 2012, at 6:41 PM, Aristedes Maniatis wrote:
>
>> On 14/03/12 5:59 AM, Joe Baldwin wrote:
>>> OK, I think that I may have run into this before. The ultimate "solution" was to increase memory - however, I am concerned that may have been a quick-fix and not a long-term fix.
>>>
>>> The problem is out of memory errors associated with tomcat heap.
>>
>> Under FreeBSD I had odd memory issues with tomcat 7 until I switched from using the Diablo (Sun approved) JDK and instead used the openjdk6. Then all the memory leak issues went away.
>>
>> Mind you I am still stuck with other issues to do with parallel deployment in tomcat7 but I don't think they have anything to do with Cayenne.
>>
>> Ari
>>
>>
>>
>>
>> --
>> -------------------------->
>> Aristedes Maniatis
>> GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
>
Re: Memory Usage Problem with Tomcat
Posted by Joe Baldwin <jf...@earthlink.net>.
Thanks.
Yes, it is hard to tell where to start investigating. I had a theory that if I could log when the Cayenne data objects were being created and destroyed, then I could definitively say whether this is a Cayenne related issue or not.
Joe
On Mar 13, 2012, at 6:41 PM, Aristedes Maniatis wrote:
> On 14/03/12 5:59 AM, Joe Baldwin wrote:
>> OK, I think that I may have run into this before. The ultimate "solution" was to increase memory - however, I am concerned that may have been a quick-fix and not a long-term fix.
>>
>> The problem is out of memory errors associated with tomcat heap.
>
> Under FreeBSD I had odd memory issues with tomcat 7 until I switched from using the Diablo (Sun approved) JDK and instead used the openjdk6. Then all the memory leak issues went away.
>
> Mind you I am still stuck with other issues to do with parallel deployment in tomcat7 but I don't think they have anything to do with Cayenne.
>
> Ari
>
>
>
>
> --
> -------------------------->
> Aristedes Maniatis
> GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
Re: Memory Usage Problem with Tomcat
Posted by Aristedes Maniatis <ar...@maniatis.org>.
On 14/03/12 5:59 AM, Joe Baldwin wrote:
> OK, I think that I may have run into this before. The ultimate "solution" was to increase memory - however, I am concerned that may have been a quick-fix and not a long-term fix.
>
> The problem is out of memory errors associated with tomcat heap.
Under FreeBSD I had odd memory issues with tomcat 7 until I switched from using the Diablo (Sun approved) JDK and instead used the openjdk6. Then all the memory leak issues went away.
Mind you I am still stuck with other issues to do with parallel deployment in tomcat7 but I don't think they have anything to do with Cayenne.
Ari
--
-------------------------->
Aristedes Maniatis
GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
Re: Memory Usage Problem with Tomcat
Posted by Andrew Lindesay <ap...@lindesay.co.nz>.
Hi Joe;
I wonder; is there a possibility that the deployment environment is
re-distributing the sessions?
Regards;
On 14/03/12 7:59 AM, Joe Baldwin wrote:
> OK, I think that I may have run into this before. The ultimate "solution" was to increase memory - however, I am concerned that may have been a quick-fix and not a long-term fix.
>
> The problem is out of memory errors associated with tomcat heap.
>
> I have a webapp (powered primarily by cayenne). The database has *very* little in it. I am essentially serving data (via cayenne& tomcat) and images (via tomcat).
>
> I have a private tomcat instance running on a webhost in a "shared" environment. What this means is that I *absolutely* cannot attach a profiler.
>
> I am being told by the webhost IT people (who are not always accurate in their objectivity) that my app is leaking memory (badly), and that is what caused tomcat to crash.
>
> My intuition tells me that with almost no activity on the website (because it is not live yet) and Cayenne memory management that I should be able to manage memory well, but it is not the case.
>
> So, if my goal is to determine what the problem is, and if I simply increase heap size, won't I just be masking a potential problem? i.e. if the app runs fine for a while, then mysteriously causes tomcat to run out of heap space, then couldn't there be a memory leak?
>
> If there is a memory leak, and I don't see it on my development server, and I *can't* use a profiler on my webhost, then how do I get visibility into the memory usage?
>
> Thanks
> Joe
>
>
--
Andrew Lindesay
www.silvereye.co.nz
Re: Memory Usage Problem with Tomcat
Posted by Andrew Lindesay <ap...@lindesay.co.nz>.
Hi Mike;
That was me; when the session carrying an object context is shifted from
container to container, with 3.0.2, I think you will find that it
consumes an exponentially growing quantity of memory. See CAY-1668.
Regards;
> 2) Modern app servers restart and redeploy applications without
> restarting the app server. Thus, the memory leak might be from a
> previous application instance or application deployment. I think
> someone reported a possible Cayenne issue for that recently.
--
Andrew Lindesay
www.silvereye.co.nz
Re: Memory Usage Problem with Tomcat
Posted by Joe Baldwin <jf...@earthlink.net>.
Excellent video, I am studying it now.
On Mar 13, 2012, at 4:47 PM, Andrus Adamchik wrote:
>
> On Mar 13, 2012, at 4:03 PM, Mike Kienenberger wrote:
>
>> 2) Modern app servers restart and redeploy applications without
>> restarting the app server. Thus, the memory leak might be from a
>> previous application instance or application deployment. I think
>> someone reported a possible Cayenne issue for that recently.
>
> I keep recommending to people this presentation by Mark Thomas from Tomcat project:
>
> Video with slides: http://www.infoq.com/presentations/Diagnosing-Memory-Leaks
> Slides in PDF: people.apache.org/~markt/presentations/2010-11-04-Memory-Leaks-60mins.pdf
>
> Even if you are not using Tomcat, but curious what happens to your memory, I still recommend it :) It is applicable to any Java app server and was an eye opener to me back in the day.
>
> Andrus
>
Re: Memory Usage Problem with Tomcat
Posted by Joe Baldwin <jf...@earthlink.net>.
Well it turns out that I executed the test incorrectly. I performed the test a second time, and I was left with superfluous Cayenne objects.
I guess that tomcat 7 isn't catching these leaks after all - which is odd, because this sounds exactly like the scenarios that Thomas was discussing in his lecture.
On Mar 15, 2012, at 4:43 PM, Andrus Adamchik wrote:
>
> On Mar 15, 2012, at 3:21 PM, Joe Baldwin wrote:
>
>> 1. what is nominal behavior for Cayenne 3.0.1 - wrt the EventManager Dispatch Thread? Should there be 5 of them normally or should there only be one.
>
> 5 dispatch threads is the default. So this is correct and means that you only have 1 Cayenne stack and 1 EventManager in the app.
>
> Andrus
Re: Memory Usage Problem with Tomcat
Posted by Joe Baldwin <jf...@earthlink.net>.
Andrus,
I am going to need your help devising another test, after the one I just performed. I am not 100% I devised it correctly (since I screwed up one yesterday).
I just copied my entire project to the Tomcat7 webapps dir, but then deleted *all* of my JSP files, and my project jar file. Essentially all that was left was some of the standard jar files in the WEB-INF/lib dir and my web.xml file (with the cayenne filter). I then started up tomcat 7, attached visualvm, then undeployed then app using tomcat manager, and then took a vvm snapshot, and the 5 Cayenne EventManagers are still there (after the only web app deployed with cayenne has been undeployed)
Please look at the included info below, and let me know if I made a bad assumption (that assumption being that what is left is standard libraries, and tomcat 7). Oh, yes I still have some definitions in the web.xml file that are jsp dead ends.
Please let me know what I can do next.
Joe
Here are the libs left in my lib dir
-rw-r--r-- 1 root wheel 51841 Mar 16 20:32 CMSMap.map.xml
-rw-r--r-- 1 root wheel 139574 Mar 16 20:32 ashwood-2.0.jar
-rw-r--r-- 1 root wheel 2046141 Mar 16 20:32 cayenne-server-3.0.1.jar
-rw-r--r-- 1 root wheel 324 Mar 16 20:32 cayenne.xml
-rw-r--r-- 1 root wheel 559366 Mar 16 20:32 commons-collections-3.1.jar
-rw-r--r-- 1 root wheel 160519 Mar 16 20:32 commons-dbcp-1.4.jar
-rw-r--r-- 1 root wheel 59590 Mar 16 20:32 commons-fileupload-1.2.2.jar
-rw-r--r-- 1 root wheel 159509 Mar 16 20:32 commons-io-2.0.1.jar
-rw-r--r-- 1 root wheel 52915 Mar 16 20:32 commons-logging-1.1.jar
-rw-r--r-- 1 root wheel 100472 Mar 16 20:32 commons-pool-1.5.6.jar
-rw-r--r-- 1 root wheel 566 Mar 16 20:32 dbcp.properties
-rw-r--r-- 1 root wheel 46047 Mar 16 20:32 jsr173_1.0_api.jar
-rw-r--r-- 1 root wheel 850 Mar 16 20:32 logging.properties
-rw-r--r-- 1 root wheel 494975 Mar 16 20:32 mail.jar
-rw-r--r-- 1 root wheel 529876 Mar 16 20:32 vpp-2.2.1.jar
here is the cayenne filter designation
<filter>
<filter-name>CayenneFilter</filter-name>
<filter-class>org.apache.cayenne.conf.WebApplicationContextFilter</filter-class>
</filter>
<filter-mapping>
<filter-name>CayenneFilter</filter-name>
<url-pattern>/*</url-pattern>
</filter-mapping>
On Mar 15, 2012, at 4:43 PM, Andrus Adamchik wrote:
>
> On Mar 15, 2012, at 3:21 PM, Joe Baldwin wrote:
>
>> 1. what is nominal behavior for Cayenne 3.0.1 - wrt the EventManager Dispatch Thread? Should there be 5 of them normally or should there only be one.
>
> 5 dispatch threads is the default. So this is correct and means that you only have 1 Cayenne stack and 1 EventManager in the app.
>
> Andrus
Re: Memory Usage Problem with Tomcat
Posted by Joe Baldwin <jf...@earthlink.net>.
Ari,
No I am not using parallel deployment (although it does sound like an interesting feature of Tomcat). In fact, my methodology, is to simplify the app, config, and lib count so that I can reproduce the problem at its most essential.
I have been eliminating library after library and the leak still appears to persist. I am waiting to hear back as to whether I have made some sort of bad assumption in devising my test.
But the short answer to your question, is that I am basically using tomcat 7 with the standard default configurations. I still have not been able to determine if this includes leak protection (that was added to tomcat 7). If it does then this problem is even more perplexing.
Joe
On Mar 17, 2012, at 6:30 AM, Aristedes Maniatis wrote:
> Are you using the Tomcat7 parallel deployment mechanism? I am finding some unusual behaviour there with the older applications not being properly stopped.
>
> Ari
>
>
> On 17/03/12 11:25 AM, Joe Baldwin wrote:
>> Andrus,
>>
>> I just performed an interesting test.
>>
>> 1. installed my web app into the tomcat 7 webapps dir
>> 2. however, this time, I deleted my app's jar file (that contains all of my Cayenne Persistence Objects created by the modeler)
>> 2. started tomcat 7
>> 3. the visualvm snapshot showed the nominal 5 Cayenne EventManagers
>> 4. I then undeployed the webapp with Tomcat 7 Manager
>> 5. I then took a new visualvm snapshot that showed the same 5 Cayenne EventManagers
>>
>> I have 0 files in the WEB-INF/classes dir. The only other classes would be the JSP's - and I believe that I deleted the entire cache correctly before I started Tomcat7. So I am *fairly* sure I have eliminated my code - well unless I am wrong abut how tomcat handles the JSP's.
>>
>> This looks very odd to me. I do have quite a few libraries that are in my WEB-INF/lib dir. I wonder if there is some sort of interaction.
>>
>> Please let me know if you have ideas for better tests. (I think I am going to try to eliminate all of my jsp's and then the other apache libs.
>>
>> Joe
>>
>>
>>
>> On Mar 15, 2012, at 4:43 PM, Andrus Adamchik wrote:
>>
>>>
>>> On Mar 15, 2012, at 3:21 PM, Joe Baldwin wrote:
>>>
>>>> 1. what is nominal behavior for Cayenne 3.0.1 - wrt the EventManager Dispatch Thread? Should there be 5 of them normally or should there only be one.
>>>
>>> 5 dispatch threads is the default. So this is correct and means that you only have 1 Cayenne stack and 1 EventManager in the app.
>>>
>>> Andrus
>>
>
> --
> -------------------------->
> Aristedes Maniatis
> GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
Re: Memory Usage Problem with Tomcat
Posted by Aristedes Maniatis <ar...@maniatis.org>.
Are you using the Tomcat7 parallel deployment mechanism? I am finding some unusual behaviour there with the older applications not being properly stopped.
Ari
On 17/03/12 11:25 AM, Joe Baldwin wrote:
> Andrus,
>
> I just performed an interesting test.
>
> 1. installed my web app into the tomcat 7 webapps dir
> 2. however, this time, I deleted my app's jar file (that contains all of my Cayenne Persistence Objects created by the modeler)
> 2. started tomcat 7
> 3. the visualvm snapshot showed the nominal 5 Cayenne EventManagers
> 4. I then undeployed the webapp with Tomcat 7 Manager
> 5. I then took a new visualvm snapshot that showed the same 5 Cayenne EventManagers
>
> I have 0 files in the WEB-INF/classes dir. The only other classes would be the JSP's - and I believe that I deleted the entire cache correctly before I started Tomcat7. So I am *fairly* sure I have eliminated my code - well unless I am wrong abut how tomcat handles the JSP's.
>
> This looks very odd to me. I do have quite a few libraries that are in my WEB-INF/lib dir. I wonder if there is some sort of interaction.
>
> Please let me know if you have ideas for better tests. (I think I am going to try to eliminate all of my jsp's and then the other apache libs.
>
> Joe
>
>
>
> On Mar 15, 2012, at 4:43 PM, Andrus Adamchik wrote:
>
>>
>> On Mar 15, 2012, at 3:21 PM, Joe Baldwin wrote:
>>
>>> 1. what is nominal behavior for Cayenne 3.0.1 - wrt the EventManager Dispatch Thread? Should there be 5 of them normally or should there only be one.
>>
>> 5 dispatch threads is the default. So this is correct and means that you only have 1 Cayenne stack and 1 EventManager in the app.
>>
>> Andrus
>
--
-------------------------->
Aristedes Maniatis
GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
Re: Memory Usage Problem with Tomcat
Posted by Joe Baldwin <jf...@earthlink.net>.
Andrus,
I just performed an interesting test.
1. installed my web app into the tomcat 7 webapps dir
2. however, this time, I deleted my app's jar file (that contains all of my Cayenne Persistence Objects created by the modeler)
2. started tomcat 7
3. the visualvm snapshot showed the nominal 5 Cayenne EventManagers
4. I then undeployed the webapp with Tomcat 7 Manager
5. I then took a new visualvm snapshot that showed the same 5 Cayenne EventManagers
I have 0 files in the WEB-INF/classes dir. The only other classes would be the JSP's - and I believe that I deleted the entire cache correctly before I started Tomcat7. So I am *fairly* sure I have eliminated my code - well unless I am wrong abut how tomcat handles the JSP's.
This looks very odd to me. I do have quite a few libraries that are in my WEB-INF/lib dir. I wonder if there is some sort of interaction.
Please let me know if you have ideas for better tests. (I think I am going to try to eliminate all of my jsp's and then the other apache libs.
Joe
On Mar 15, 2012, at 4:43 PM, Andrus Adamchik wrote:
>
> On Mar 15, 2012, at 3:21 PM, Joe Baldwin wrote:
>
>> 1. what is nominal behavior for Cayenne 3.0.1 - wrt the EventManager Dispatch Thread? Should there be 5 of them normally or should there only be one.
>
> 5 dispatch threads is the default. So this is correct and means that you only have 1 Cayenne stack and 1 EventManager in the app.
>
> Andrus
Re: Memory Usage Problem with Tomcat
Posted by Andrus Adamchik <an...@objectstyle.org>.
On Mar 15, 2012, at 3:21 PM, Joe Baldwin wrote:
> 1. what is nominal behavior for Cayenne 3.0.1 - wrt the EventManager Dispatch Thread? Should there be 5 of them normally or should there only be one.
5 dispatch threads is the default. So this is correct and means that you only have 1 Cayenne stack and 1 EventManager in the app.
Andrus
Re: Memory Usage Problem with Tomcat
Posted by Joe Baldwin <jf...@earthlink.net>.
Andrus,
Thomas spoke of some fixes for the classloader (among a great many other issues) in Tomcat 7, so I installed it (which is not nearly as straighforward as Tomcat 6 BTW), and guess what ... it only has the 5 instances of the EventManger.
So here is my question list:
1. what is nominal behavior for Cayenne 3.0.1 - wrt the EventManager Dispatch Thread? Should there be 5 of them normally or should there only be one.
2. it *appears* that the EventManger is being GC'ed correctly (as far as I can tell) with Tomcat 7. However, this may be due to the additional Classloader leak prevention feature (if it is in fact turned on by default). Do you agree - or are the 5 EventManagers a problem as well?
Thanks
Joe
On Mar 14, 2012, at 3:41 PM, Andrus Adamchik wrote:
>
> On Mar 14, 2012, at 2:09 PM, Joe Baldwin wrote:
>
>> Almost forgot:
>>
>> I am using 3.0.1 because it was recommended by your team as a production release. Do I need to upgrade?
>
> The issue CAY-1467 that sounds very similar to your problem was fixed in 3.0.1 (at least according to the RELEASE-NOTES.txt). You may try upgrading to 3.0.2 just in case (which is a drop-in replacement). But this should be already in 3.0.1.
>
> Andrus
Re: Memory Usage Problem with Tomcat
Posted by Andrus Adamchik <an...@objectstyle.org>.
On Mar 14, 2012, at 2:09 PM, Joe Baldwin wrote:
> Almost forgot:
>
> I am using 3.0.1 because it was recommended by your team as a production release. Do I need to upgrade?
The issue CAY-1467 that sounds very similar to your problem was fixed in 3.0.1 (at least according to the RELEASE-NOTES.txt). You may try upgrading to 3.0.2 just in case (which is a drop-in replacement). But this should be already in 3.0.1.
Andrus
Re: Memory Usage Problem with Tomcat
Posted by Joe Baldwin <jf...@earthlink.net>.
Almost forgot:
I am using 3.0.1 because it was recommended by your team as a production release. Do I need to upgrade?
On Mar 14, 2012, at 1:44 PM, Andrus Adamchik wrote:
> I didn't know how you start/stop Cayenne so I was giving a general answer. If you are using WebApplicationContextFilter, it should stop Cayenne event threads. Maybe you can run your web app in debugger locally, put a breakpoint in WebApplicationContextFilter.destroy() and see if it is invoked at all, and when it is called - that the threads stop.
>
> Andrus
>
>
> On Mar 14, 2012, at 12:43 PM, Joe Baldwin wrote:
>
>> Andrus,
>>
>>> You will need something similar.
>>
>>
>> I was following you until you said this. The link you sent is code for the cayenne filter that is intended to be placed in the web.xml file for tomcat. I have already done this. So is your comment meant to imply that I need to *add* the filter, or *write* one myself.
>>
>> Whats more, if I have implemented your filter, then why would I need to create another one.
>>
>> Also, are these objects that hold on to my data objects? If so, then this is a rather important issue.
>>
>> Thanks
>> Joe
>>
>>
>>
>> On Mar 14, 2012, at 9:59 AM, Andrus Adamchik wrote:
>>
>>> Ok. Now you are on to something. The remaining Cayenne event threads is an indicator that the old version of the app was not undeployed.
>>>
>>> You need to ensure that you shut down your Cayenne Configuration (in 3.0) or ServerRuntime (in 3.1) when your web application stops. Here is an example of how Cayenne does it in WebApplicationContextFilter.destroy() :
>>>
>>> http://svn.apache.org/viewvc/cayenne/main/branches/STABLE-3.0/framework/cayenne-jdk1.5-unpublished/src/main/java/org/apache/cayenne/conf/WebApplicationContextFilter.java?view=markup
>>>
>>> You will need something similar.
>>>
>>> Andrus
>>>
>>>
>>> On Mar 14, 2012, at 9:51 AM, Joe Baldwin wrote:
>>>
>>>> Andrus,
>>>>
>>>> After watching the video you suggested, I tried one of the "tricks". I started up tomcat, with the cayenne enabled app. Then took a snapshot with VisualVM. There appear to be 5 instances of
>>>> org.apache.cayenne.event.EventManager$DispatchThread.run()
>>>>
>>>> This may or may not be normal, however when I redeployed the application (via ant-tomcat command) there were then 10 instances. Each time I redeployed the app, another 5 instances were displayed with VisualVM snapshot.
>>>>
>>>> I then tried another test, in which I restarted Tomcat (to get 5 instances of the EventManager) used Tomcat manager to undeploy the app (which ostensibly wipes the app from the tomcat dir structure), however, the 5 instances of the EventManager were still showing up in VisualVM snapshot.
>>>>
>>>> Mark Thomas (the video lecturer), said that this all *might* indicate a leak. However, since I am unfamiliar with the intended behavior, I cannot be sure. On the other hand, having these 5 instance multiply each time the app is redeployed does not seem like it is standard behavior.
>>>>
>>>> Thomas described an on behavior in which just adding a JDBC jar file to your WEB-INF/lib dir could cause a leak. The reasoning behind how the class loaders work was a bit convoluted, but I am wondering if these behaviors are related.
>>>>
>>>> So my question is what behavior would you expect, and what should my next test be?
>>>>
>>>> Joe
>>>>
>>>>
>>>> On Mar 13, 2012, at 4:47 PM, Andrus Adamchik wrote:
>>>>
>>>>>
>>>>> On Mar 13, 2012, at 4:03 PM, Mike Kienenberger wrote:
>>>>>
>>>>>> 2) Modern app servers restart and redeploy applications without
>>>>>> restarting the app server. Thus, the memory leak might be from a
>>>>>> previous application instance or application deployment. I think
>>>>>> someone reported a possible Cayenne issue for that recently.
>>>>>
>>>>> I keep recommending to people this presentation by Mark Thomas from Tomcat project:
>>>>>
>>>>> Video with slides: http://www.infoq.com/presentations/Diagnosing-Memory-Leaks
>>>>> Slides in PDF: people.apache.org/~markt/presentations/2010-11-04-Memory-Leaks-60mins.pdf
>>>>>
>>>>> Even if you are not using Tomcat, but curious what happens to your memory, I still recommend it :) It is applicable to any Java app server and was an eye opener to me back in the day.
>>>>>
>>>>> Andrus
>>>>>
>>>>
>>>>
>>>
>>
>>
>
Re: Memory Usage Problem with Tomcat
Posted by Andrus Adamchik <an...@objectstyle.org>.
On Mar 14, 2012, at 2:08 PM, Joe Baldwin wrote:
> I am one of the few people who thinks Eclipse is annoying.
Eclipse has a built in debugger. This is what I am using.
Andrus
Re: Memory Usage Problem with Tomcat
Posted by Joe Baldwin <jf...@earthlink.net>.
I do not want this to degrade into a "my debugger" vs "your debugger" debate, since I have about 4 high priority items to solve in the next few days.
I have lots of experience with different languages, including C, C++, Objective-C, Java, JSP, Javascript, jQuery, on Sun, Silicon Graphics, Windoz, OSX, Linux. I have used everything from visual debuggers to command line debuggers. In a project like the one I am working on, with many different languages sitting on top of each other, and with browsers causing their on brand of problems, I have made a very considered decisions that "less is better" when trying to debug problems that may be due to everything from a single library to a combination of competing technologies.
If I were making the kinds of decisions that the Cayenne team is making, I would not hesitate to use Eclipse. However, I find it top-heavy and clumsy for the multi-level development I am doing (right up to the point someone has tested something like VisualVM that makes my life easier).
I have done a *lot* of black box testing and debuggers are a useful tool for a certain class of problem. However, for other classes of problems they create a "you can't see the forest for the trees" phenomenon.
I will give a couple of debuggers a shot, but my intuition tells me that this is a simple configuration issue at some level.
On Mar 14, 2012, at 3:05 PM, John Huss wrote:
> On Wed, Mar 14, 2012 at 1:08 PM, Joe Baldwin <jf...@earthlink.net>wrote:
>
>> Andrus,
>>
>> that is a rather large amount of work, since I use TextWrangler Editor
>> (i.e. no IDE and not debugger). If you have a recommendation for an
>> opensource debugger, then I will give it a shot. (I am one of the few
>> people who thinks Eclipse is annoying. :) )
>
>
> Not being able to debug your code is far more annoying. However, though I
> have never used it, java comes with a command-line debugger - jdb.
>
> John
Re: Memory Usage Problem with Tomcat
Posted by John Huss <jo...@gmail.com>.
On Wed, Mar 14, 2012 at 1:08 PM, Joe Baldwin <jf...@earthlink.net>wrote:
> Andrus,
>
> that is a rather large amount of work, since I use TextWrangler Editor
> (i.e. no IDE and not debugger). If you have a recommendation for an
> opensource debugger, then I will give it a shot. (I am one of the few
> people who thinks Eclipse is annoying. :) )
Not being able to debug your code is far more annoying. However, though I
have never used it, java comes with a command-line debugger - jdb.
John
Re: Memory Usage Problem with Tomcat
Posted by Joe Baldwin <jf...@earthlink.net>.
Andrus,
that is a rather large amount of work, since I use TextWrangler Editor (i.e. no IDE and not debugger). If you have a recommendation for an opensource debugger, then I will give it a shot. (I am one of the few people who thinks Eclipse is annoying. :) )
In the interim, here is my web.xml code (which I added a *long* time ago - which appears to me to be the current methodology)
<filter>
<filter-name>CayenneFilter</filter-name>
<filter-class>org.apache.cayenne.conf.WebApplicationContextFilter</filter-class>
</filter>
<filter-mapping>
<filter-name>CayenneFilter</filter-name>
<url-pattern>/*</url-pattern>
</filter-mapping>
What in your experience would lead to the filter destroy method not being called? Is there another library that could be holding on to a reference?
Thanks,
Joe
On Mar 14, 2012, at 1:44 PM, Andrus Adamchik wrote:
> I didn't know how you start/stop Cayenne so I was giving a general answer. If you are using WebApplicationContextFilter, it should stop Cayenne event threads. Maybe you can run your web app in debugger locally, put a breakpoint in WebApplicationContextFilter.destroy() and see if it is invoked at all, and when it is called - that the threads stop.
>
> Andrus
>
>
> On Mar 14, 2012, at 12:43 PM, Joe Baldwin wrote:
>
>> Andrus,
>>
>>> You will need something similar.
>>
>>
>> I was following you until you said this. The link you sent is code for the cayenne filter that is intended to be placed in the web.xml file for tomcat. I have already done this. So is your comment meant to imply that I need to *add* the filter, or *write* one myself.
>>
>> Whats more, if I have implemented your filter, then why would I need to create another one.
>>
>> Also, are these objects that hold on to my data objects? If so, then this is a rather important issue.
>>
>> Thanks
>> Joe
>>
>>
>>
>> On Mar 14, 2012, at 9:59 AM, Andrus Adamchik wrote:
>>
>>> Ok. Now you are on to something. The remaining Cayenne event threads is an indicator that the old version of the app was not undeployed.
>>>
>>> You need to ensure that you shut down your Cayenne Configuration (in 3.0) or ServerRuntime (in 3.1) when your web application stops. Here is an example of how Cayenne does it in WebApplicationContextFilter.destroy() :
>>>
>>> http://svn.apache.org/viewvc/cayenne/main/branches/STABLE-3.0/framework/cayenne-jdk1.5-unpublished/src/main/java/org/apache/cayenne/conf/WebApplicationContextFilter.java?view=markup
>>>
>>> You will need something similar.
>>>
>>> Andrus
>>>
>>>
>>> On Mar 14, 2012, at 9:51 AM, Joe Baldwin wrote:
>>>
>>>> Andrus,
>>>>
>>>> After watching the video you suggested, I tried one of the "tricks". I started up tomcat, with the cayenne enabled app. Then took a snapshot with VisualVM. There appear to be 5 instances of
>>>> org.apache.cayenne.event.EventManager$DispatchThread.run()
>>>>
>>>> This may or may not be normal, however when I redeployed the application (via ant-tomcat command) there were then 10 instances. Each time I redeployed the app, another 5 instances were displayed with VisualVM snapshot.
>>>>
>>>> I then tried another test, in which I restarted Tomcat (to get 5 instances of the EventManager) used Tomcat manager to undeploy the app (which ostensibly wipes the app from the tomcat dir structure), however, the 5 instances of the EventManager were still showing up in VisualVM snapshot.
>>>>
>>>> Mark Thomas (the video lecturer), said that this all *might* indicate a leak. However, since I am unfamiliar with the intended behavior, I cannot be sure. On the other hand, having these 5 instance multiply each time the app is redeployed does not seem like it is standard behavior.
>>>>
>>>> Thomas described an on behavior in which just adding a JDBC jar file to your WEB-INF/lib dir could cause a leak. The reasoning behind how the class loaders work was a bit convoluted, but I am wondering if these behaviors are related.
>>>>
>>>> So my question is what behavior would you expect, and what should my next test be?
>>>>
>>>> Joe
>>>>
>>>>
>>>> On Mar 13, 2012, at 4:47 PM, Andrus Adamchik wrote:
>>>>
>>>>>
>>>>> On Mar 13, 2012, at 4:03 PM, Mike Kienenberger wrote:
>>>>>
>>>>>> 2) Modern app servers restart and redeploy applications without
>>>>>> restarting the app server. Thus, the memory leak might be from a
>>>>>> previous application instance or application deployment. I think
>>>>>> someone reported a possible Cayenne issue for that recently.
>>>>>
>>>>> I keep recommending to people this presentation by Mark Thomas from Tomcat project:
>>>>>
>>>>> Video with slides: http://www.infoq.com/presentations/Diagnosing-Memory-Leaks
>>>>> Slides in PDF: people.apache.org/~markt/presentations/2010-11-04-Memory-Leaks-60mins.pdf
>>>>>
>>>>> Even if you are not using Tomcat, but curious what happens to your memory, I still recommend it :) It is applicable to any Java app server and was an eye opener to me back in the day.
>>>>>
>>>>> Andrus
>>>>>
>>>>
>>>>
>>>
>>
>>
>
Re: Memory Usage Problem with Tomcat
Posted by Andrus Adamchik <an...@objectstyle.org>.
I didn't know how you start/stop Cayenne so I was giving a general answer. If you are using WebApplicationContextFilter, it should stop Cayenne event threads. Maybe you can run your web app in debugger locally, put a breakpoint in WebApplicationContextFilter.destroy() and see if it is invoked at all, and when it is called - that the threads stop.
Andrus
On Mar 14, 2012, at 12:43 PM, Joe Baldwin wrote:
> Andrus,
>
>> You will need something similar.
>
>
> I was following you until you said this. The link you sent is code for the cayenne filter that is intended to be placed in the web.xml file for tomcat. I have already done this. So is your comment meant to imply that I need to *add* the filter, or *write* one myself.
>
> Whats more, if I have implemented your filter, then why would I need to create another one.
>
> Also, are these objects that hold on to my data objects? If so, then this is a rather important issue.
>
> Thanks
> Joe
>
>
>
> On Mar 14, 2012, at 9:59 AM, Andrus Adamchik wrote:
>
>> Ok. Now you are on to something. The remaining Cayenne event threads is an indicator that the old version of the app was not undeployed.
>>
>> You need to ensure that you shut down your Cayenne Configuration (in 3.0) or ServerRuntime (in 3.1) when your web application stops. Here is an example of how Cayenne does it in WebApplicationContextFilter.destroy() :
>>
>> http://svn.apache.org/viewvc/cayenne/main/branches/STABLE-3.0/framework/cayenne-jdk1.5-unpublished/src/main/java/org/apache/cayenne/conf/WebApplicationContextFilter.java?view=markup
>>
>> You will need something similar.
>>
>> Andrus
>>
>>
>> On Mar 14, 2012, at 9:51 AM, Joe Baldwin wrote:
>>
>>> Andrus,
>>>
>>> After watching the video you suggested, I tried one of the "tricks". I started up tomcat, with the cayenne enabled app. Then took a snapshot with VisualVM. There appear to be 5 instances of
>>> org.apache.cayenne.event.EventManager$DispatchThread.run()
>>>
>>> This may or may not be normal, however when I redeployed the application (via ant-tomcat command) there were then 10 instances. Each time I redeployed the app, another 5 instances were displayed with VisualVM snapshot.
>>>
>>> I then tried another test, in which I restarted Tomcat (to get 5 instances of the EventManager) used Tomcat manager to undeploy the app (which ostensibly wipes the app from the tomcat dir structure), however, the 5 instances of the EventManager were still showing up in VisualVM snapshot.
>>>
>>> Mark Thomas (the video lecturer), said that this all *might* indicate a leak. However, since I am unfamiliar with the intended behavior, I cannot be sure. On the other hand, having these 5 instance multiply each time the app is redeployed does not seem like it is standard behavior.
>>>
>>> Thomas described an on behavior in which just adding a JDBC jar file to your WEB-INF/lib dir could cause a leak. The reasoning behind how the class loaders work was a bit convoluted, but I am wondering if these behaviors are related.
>>>
>>> So my question is what behavior would you expect, and what should my next test be?
>>>
>>> Joe
>>>
>>>
>>> On Mar 13, 2012, at 4:47 PM, Andrus Adamchik wrote:
>>>
>>>>
>>>> On Mar 13, 2012, at 4:03 PM, Mike Kienenberger wrote:
>>>>
>>>>> 2) Modern app servers restart and redeploy applications without
>>>>> restarting the app server. Thus, the memory leak might be from a
>>>>> previous application instance or application deployment. I think
>>>>> someone reported a possible Cayenne issue for that recently.
>>>>
>>>> I keep recommending to people this presentation by Mark Thomas from Tomcat project:
>>>>
>>>> Video with slides: http://www.infoq.com/presentations/Diagnosing-Memory-Leaks
>>>> Slides in PDF: people.apache.org/~markt/presentations/2010-11-04-Memory-Leaks-60mins.pdf
>>>>
>>>> Even if you are not using Tomcat, but curious what happens to your memory, I still recommend it :) It is applicable to any Java app server and was an eye opener to me back in the day.
>>>>
>>>> Andrus
>>>>
>>>
>>>
>>
>
>
Re: Memory Usage Problem with Tomcat
Posted by Joe Baldwin <jf...@earthlink.net>.
Andrus,
> You will need something similar.
I was following you until you said this. The link you sent is code for the cayenne filter that is intended to be placed in the web.xml file for tomcat. I have already done this. So is your comment meant to imply that I need to *add* the filter, or *write* one myself.
Whats more, if I have implemented your filter, then why would I need to create another one.
Also, are these objects that hold on to my data objects? If so, then this is a rather important issue.
Thanks
Joe
On Mar 14, 2012, at 9:59 AM, Andrus Adamchik wrote:
> Ok. Now you are on to something. The remaining Cayenne event threads is an indicator that the old version of the app was not undeployed.
>
> You need to ensure that you shut down your Cayenne Configuration (in 3.0) or ServerRuntime (in 3.1) when your web application stops. Here is an example of how Cayenne does it in WebApplicationContextFilter.destroy() :
>
> http://svn.apache.org/viewvc/cayenne/main/branches/STABLE-3.0/framework/cayenne-jdk1.5-unpublished/src/main/java/org/apache/cayenne/conf/WebApplicationContextFilter.java?view=markup
>
> You will need something similar.
>
> Andrus
>
>
> On Mar 14, 2012, at 9:51 AM, Joe Baldwin wrote:
>
>> Andrus,
>>
>> After watching the video you suggested, I tried one of the "tricks". I started up tomcat, with the cayenne enabled app. Then took a snapshot with VisualVM. There appear to be 5 instances of
>> org.apache.cayenne.event.EventManager$DispatchThread.run()
>>
>> This may or may not be normal, however when I redeployed the application (via ant-tomcat command) there were then 10 instances. Each time I redeployed the app, another 5 instances were displayed with VisualVM snapshot.
>>
>> I then tried another test, in which I restarted Tomcat (to get 5 instances of the EventManager) used Tomcat manager to undeploy the app (which ostensibly wipes the app from the tomcat dir structure), however, the 5 instances of the EventManager were still showing up in VisualVM snapshot.
>>
>> Mark Thomas (the video lecturer), said that this all *might* indicate a leak. However, since I am unfamiliar with the intended behavior, I cannot be sure. On the other hand, having these 5 instance multiply each time the app is redeployed does not seem like it is standard behavior.
>>
>> Thomas described an on behavior in which just adding a JDBC jar file to your WEB-INF/lib dir could cause a leak. The reasoning behind how the class loaders work was a bit convoluted, but I am wondering if these behaviors are related.
>>
>> So my question is what behavior would you expect, and what should my next test be?
>>
>> Joe
>>
>>
>> On Mar 13, 2012, at 4:47 PM, Andrus Adamchik wrote:
>>
>>>
>>> On Mar 13, 2012, at 4:03 PM, Mike Kienenberger wrote:
>>>
>>>> 2) Modern app servers restart and redeploy applications without
>>>> restarting the app server. Thus, the memory leak might be from a
>>>> previous application instance or application deployment. I think
>>>> someone reported a possible Cayenne issue for that recently.
>>>
>>> I keep recommending to people this presentation by Mark Thomas from Tomcat project:
>>>
>>> Video with slides: http://www.infoq.com/presentations/Diagnosing-Memory-Leaks
>>> Slides in PDF: people.apache.org/~markt/presentations/2010-11-04-Memory-Leaks-60mins.pdf
>>>
>>> Even if you are not using Tomcat, but curious what happens to your memory, I still recommend it :) It is applicable to any Java app server and was an eye opener to me back in the day.
>>>
>>> Andrus
>>>
>>
>>
>
Re: Memory Usage Problem with Tomcat
Posted by Andrus Adamchik <an...@objectstyle.org>.
Ok. Now you are on to something. The remaining Cayenne event threads is an indicator that the old version of the app was not undeployed.
You need to ensure that you shut down your Cayenne Configuration (in 3.0) or ServerRuntime (in 3.1) when your web application stops. Here is an example of how Cayenne does it in WebApplicationContextFilter.destroy() :
http://svn.apache.org/viewvc/cayenne/main/branches/STABLE-3.0/framework/cayenne-jdk1.5-unpublished/src/main/java/org/apache/cayenne/conf/WebApplicationContextFilter.java?view=markup
You will need something similar.
Andrus
On Mar 14, 2012, at 9:51 AM, Joe Baldwin wrote:
> Andrus,
>
> After watching the video you suggested, I tried one of the "tricks". I started up tomcat, with the cayenne enabled app. Then took a snapshot with VisualVM. There appear to be 5 instances of
> org.apache.cayenne.event.EventManager$DispatchThread.run()
>
> This may or may not be normal, however when I redeployed the application (via ant-tomcat command) there were then 10 instances. Each time I redeployed the app, another 5 instances were displayed with VisualVM snapshot.
>
> I then tried another test, in which I restarted Tomcat (to get 5 instances of the EventManager) used Tomcat manager to undeploy the app (which ostensibly wipes the app from the tomcat dir structure), however, the 5 instances of the EventManager were still showing up in VisualVM snapshot.
>
> Mark Thomas (the video lecturer), said that this all *might* indicate a leak. However, since I am unfamiliar with the intended behavior, I cannot be sure. On the other hand, having these 5 instance multiply each time the app is redeployed does not seem like it is standard behavior.
>
> Thomas described an on behavior in which just adding a JDBC jar file to your WEB-INF/lib dir could cause a leak. The reasoning behind how the class loaders work was a bit convoluted, but I am wondering if these behaviors are related.
>
> So my question is what behavior would you expect, and what should my next test be?
>
> Joe
>
>
> On Mar 13, 2012, at 4:47 PM, Andrus Adamchik wrote:
>
>>
>> On Mar 13, 2012, at 4:03 PM, Mike Kienenberger wrote:
>>
>>> 2) Modern app servers restart and redeploy applications without
>>> restarting the app server. Thus, the memory leak might be from a
>>> previous application instance or application deployment. I think
>>> someone reported a possible Cayenne issue for that recently.
>>
>> I keep recommending to people this presentation by Mark Thomas from Tomcat project:
>>
>> Video with slides: http://www.infoq.com/presentations/Diagnosing-Memory-Leaks
>> Slides in PDF: people.apache.org/~markt/presentations/2010-11-04-Memory-Leaks-60mins.pdf
>>
>> Even if you are not using Tomcat, but curious what happens to your memory, I still recommend it :) It is applicable to any Java app server and was an eye opener to me back in the day.
>>
>> Andrus
>>
>
>
Re: Memory Usage Problem with Tomcat
Posted by Joe Baldwin <jf...@earthlink.net>.
Andrus,
After watching the video you suggested, I tried one of the "tricks". I started up tomcat, with the cayenne enabled app. Then took a snapshot with VisualVM. There appear to be 5 instances of
org.apache.cayenne.event.EventManager$DispatchThread.run()
This may or may not be normal, however when I redeployed the application (via ant-tomcat command) there were then 10 instances. Each time I redeployed the app, another 5 instances were displayed with VisualVM snapshot.
I then tried another test, in which I restarted Tomcat (to get 5 instances of the EventManager) used Tomcat manager to undeploy the app (which ostensibly wipes the app from the tomcat dir structure), however, the 5 instances of the EventManager were still showing up in VisualVM snapshot.
Mark Thomas (the video lecturer), said that this all *might* indicate a leak. However, since I am unfamiliar with the intended behavior, I cannot be sure. On the other hand, having these 5 instance multiply each time the app is redeployed does not seem like it is standard behavior.
Thomas described an on behavior in which just adding a JDBC jar file to your WEB-INF/lib dir could cause a leak. The reasoning behind how the class loaders work was a bit convoluted, but I am wondering if these behaviors are related.
So my question is what behavior would you expect, and what should my next test be?
Joe
On Mar 13, 2012, at 4:47 PM, Andrus Adamchik wrote:
>
> On Mar 13, 2012, at 4:03 PM, Mike Kienenberger wrote:
>
>> 2) Modern app servers restart and redeploy applications without
>> restarting the app server. Thus, the memory leak might be from a
>> previous application instance or application deployment. I think
>> someone reported a possible Cayenne issue for that recently.
>
> I keep recommending to people this presentation by Mark Thomas from Tomcat project:
>
> Video with slides: http://www.infoq.com/presentations/Diagnosing-Memory-Leaks
> Slides in PDF: people.apache.org/~markt/presentations/2010-11-04-Memory-Leaks-60mins.pdf
>
> Even if you are not using Tomcat, but curious what happens to your memory, I still recommend it :) It is applicable to any Java app server and was an eye opener to me back in the day.
>
> Andrus
>
Re: Memory Usage Problem with Tomcat
Posted by Joe Baldwin <jf...@earthlink.net>.
Andrus,
Mark Thomas, explains that "rule of thumb" if you see multiple reloads (which is what I do on my dev server) result in out of memory error, then the problem is *probably* but not always a memory leaks. I have seen this on my dev server. So I might actually have a memory leak somewhere.
The second thing I found is that he is using a combo of jmeter & visualvm (which is what I coincidentally downloaded earlier today).
I have not gotten through the video, but if I can simulate the outofmemory behavior on my server, then the next thing I presume is finding out where it is coming from. Have not seen enough of the video to figure this out though. :)
I might be able to crack this yet.
Joe
On Mar 13, 2012, at 4:47 PM, Andrus Adamchik wrote:
>
> On Mar 13, 2012, at 4:03 PM, Mike Kienenberger wrote:
>
>> 2) Modern app servers restart and redeploy applications without
>> restarting the app server. Thus, the memory leak might be from a
>> previous application instance or application deployment. I think
>> someone reported a possible Cayenne issue for that recently.
>
> I keep recommending to people this presentation by Mark Thomas from Tomcat project:
>
> Video with slides: http://www.infoq.com/presentations/Diagnosing-Memory-Leaks
> Slides in PDF: people.apache.org/~markt/presentations/2010-11-04-Memory-Leaks-60mins.pdf
>
> Even if you are not using Tomcat, but curious what happens to your memory, I still recommend it :) It is applicable to any Java app server and was an eye opener to me back in the day.
>
> Andrus
>
Re: Memory Usage Problem with Tomcat
Posted by Andrus Adamchik <an...@objectstyle.org>.
On Mar 13, 2012, at 4:03 PM, Mike Kienenberger wrote:
> 2) Modern app servers restart and redeploy applications without
> restarting the app server. Thus, the memory leak might be from a
> previous application instance or application deployment. I think
> someone reported a possible Cayenne issue for that recently.
I keep recommending to people this presentation by Mark Thomas from Tomcat project:
Video with slides: http://www.infoq.com/presentations/Diagnosing-Memory-Leaks
Slides in PDF: people.apache.org/~markt/presentations/2010-11-04-Memory-Leaks-60mins.pdf
Even if you are not using Tomcat, but curious what happens to your memory, I still recommend it :) It is applicable to any Java app server and was an eye opener to me back in the day.
Andrus
Re: Memory Usage Problem with Tomcat
Posted by Mike Kienenberger <mk...@gmail.com>.
It's not so much the trace part in this case as the exact error.
Sometimes the exact error will help you know what you need to look
into:
java.lang.OutOfMemoryError: Java heap space
vs
java.lang.OutOfMemoryError: PermGen space
Here are a couple things that make app server memory issues more difficult:
1) You sometimes don't know if the app is leaking memory or the app
server itself.
2) Modern app servers restart and redeploy applications without
restarting the app server. Thus, the memory leak might be from a
previous application instance or application deployment. I think
someone reported a possible Cayenne issue for that recently.
I generally try to give an app server (or Eclipse for that matter) as
much memory as is possible. Most of the time, app servers run on
dedicated machines so if you're not allowing the app server to use the
memory, it's going to waste. Obviously, that's not as easy if you
have to share memory among multiple JVM instances.
The only ways I know to determine how much memory is needed is to
actually measure it or to calculate it somehow. Most of the time,
it's far faster to just bump up the memory. At some point, if
there's a real problem, this still doesn't help and you'll have to
measure or calculate usage. I think there's an option to the Sun JVM
to log periodic memory statistics, but I might be thinking of
something else.
On Tue, Mar 13, 2012 at 3:41 PM, Joe Baldwin <jf...@earthlink.net> wrote:
> Mike,
>
> thanks for your ideas. I am still a bit mystified by the rules concerning garbage collection I worked on a a project some time back on this and became very adept at managing memory. However, when I have advanced software like Tomcat & Cayenne in between my software and the JVM, then I get a tad confused with what is going on.
>
> What I have been reading today tells me that the "rule of thumb" is to simply increase memory. However, I have two conflicting thoughts about this:
> 1. if I do, in fact, have an error in my code then increasing memory will simply hide the error for a while.
> 2. if you look at it like an operating system, if you try to run OSX on a computer with 128MB of ram, then you are insane - and no amount of correct programming will allow your app to run without running out of memory.
>
> As to your questions: the stack trace was not terribly helpful since it indicated an out of memory error when a query was launched (on a table that had all of 7 items in it).
>
> Your idea of trying to set up a similar config on my dev server may be a good idea. I probably need to take a look at the general health of the memory management. Of course, I could just be running with way too-little memory.
>
> I wonder what a good base memory setting should be: 64, 128, 512, 1Gig? The comments on the web seem to indicate all of them.
>
> Thanks
> Joe
>
>
>
> On Mar 13, 2012, at 3:19 PM, Mike Kienenberger wrote:
>
>> If you have almost no activity, why not set up a duplicate environment
>> running the same version of Tomcat and hit your application using
>> JMeter or some other testing tool?
>>
>> Or perhaps you can get sent the application http access.log file and
>> duplicate the exact series of requests that generated the problem in
>> your dev environment.
>>
>> That said, Tomcat did often seem to have intrinsic memory issues,
>> which is another reason I stopped using a few years back.
>>
>> Also, you might want to ask what the exact stacktrace is. We've had
>> situations where it was a Tomcat permgen memory issue. See this
>> article for details -- there are more details in the comments by
>> others:
>>
>> http://www.mkyong.com/tomcat/tomcat-javalangoutofmemoryerror-permgen-space/
>>
>> Again, a disclaimer as I haven't used Tomcat personally in a while,
>> although some of my colleagues continue to do so for development. And
>> we don't run app servers using it.
>>
>>
>> On Tue, Mar 13, 2012 at 2:59 PM, Joe Baldwin <jf...@earthlink.net> wrote:
>>> OK, I think that I may have run into this before. The ultimate "solution" was to increase memory - however, I am concerned that may have been a quick-fix and not a long-term fix.
>>>
>>> The problem is out of memory errors associated with tomcat heap.
>>>
>>> I have a webapp (powered primarily by cayenne). The database has *very* little in it. I am essentially serving data (via cayenne & tomcat) and images (via tomcat).
>>>
>>> I have a private tomcat instance running on a webhost in a "shared" environment. What this means is that I *absolutely* cannot attach a profiler.
>>>
>>> I am being told by the webhost IT people (who are not always accurate in their objectivity) that my app is leaking memory (badly), and that is what caused tomcat to crash.
>>>
>>> My intuition tells me that with almost no activity on the website (because it is not live yet) and Cayenne memory management that I should be able to manage memory well, but it is not the case.
>>>
>>> So, if my goal is to determine what the problem is, and if I simply increase heap size, won't I just be masking a potential problem? i.e. if the app runs fine for a while, then mysteriously causes tomcat to run out of heap space, then couldn't there be a memory leak?
>>>
>>> If there is a memory leak, and I don't see it on my development server, and I *can't* use a profiler on my webhost, then how do I get visibility into the memory usage?
>>>
>>> Thanks
>>> Joe
>>>
>
Re: Memory Usage Problem with Tomcat
Posted by Joe Baldwin <jf...@earthlink.net>.
Mike,
thanks for your ideas. I am still a bit mystified by the rules concerning garbage collection I worked on a a project some time back on this and became very adept at managing memory. However, when I have advanced software like Tomcat & Cayenne in between my software and the JVM, then I get a tad confused with what is going on.
What I have been reading today tells me that the "rule of thumb" is to simply increase memory. However, I have two conflicting thoughts about this:
1. if I do, in fact, have an error in my code then increasing memory will simply hide the error for a while.
2. if you look at it like an operating system, if you try to run OSX on a computer with 128MB of ram, then you are insane - and no amount of correct programming will allow your app to run without running out of memory.
As to your questions: the stack trace was not terribly helpful since it indicated an out of memory error when a query was launched (on a table that had all of 7 items in it).
Your idea of trying to set up a similar config on my dev server may be a good idea. I probably need to take a look at the general health of the memory management. Of course, I could just be running with way too-little memory.
I wonder what a good base memory setting should be: 64, 128, 512, 1Gig? The comments on the web seem to indicate all of them.
Thanks
Joe
On Mar 13, 2012, at 3:19 PM, Mike Kienenberger wrote:
> If you have almost no activity, why not set up a duplicate environment
> running the same version of Tomcat and hit your application using
> JMeter or some other testing tool?
>
> Or perhaps you can get sent the application http access.log file and
> duplicate the exact series of requests that generated the problem in
> your dev environment.
>
> That said, Tomcat did often seem to have intrinsic memory issues,
> which is another reason I stopped using a few years back.
>
> Also, you might want to ask what the exact stacktrace is. We've had
> situations where it was a Tomcat permgen memory issue. See this
> article for details -- there are more details in the comments by
> others:
>
> http://www.mkyong.com/tomcat/tomcat-javalangoutofmemoryerror-permgen-space/
>
> Again, a disclaimer as I haven't used Tomcat personally in a while,
> although some of my colleagues continue to do so for development. And
> we don't run app servers using it.
>
>
> On Tue, Mar 13, 2012 at 2:59 PM, Joe Baldwin <jf...@earthlink.net> wrote:
>> OK, I think that I may have run into this before. The ultimate "solution" was to increase memory - however, I am concerned that may have been a quick-fix and not a long-term fix.
>>
>> The problem is out of memory errors associated with tomcat heap.
>>
>> I have a webapp (powered primarily by cayenne). The database has *very* little in it. I am essentially serving data (via cayenne & tomcat) and images (via tomcat).
>>
>> I have a private tomcat instance running on a webhost in a "shared" environment. What this means is that I *absolutely* cannot attach a profiler.
>>
>> I am being told by the webhost IT people (who are not always accurate in their objectivity) that my app is leaking memory (badly), and that is what caused tomcat to crash.
>>
>> My intuition tells me that with almost no activity on the website (because it is not live yet) and Cayenne memory management that I should be able to manage memory well, but it is not the case.
>>
>> So, if my goal is to determine what the problem is, and if I simply increase heap size, won't I just be masking a potential problem? i.e. if the app runs fine for a while, then mysteriously causes tomcat to run out of heap space, then couldn't there be a memory leak?
>>
>> If there is a memory leak, and I don't see it on my development server, and I *can't* use a profiler on my webhost, then how do I get visibility into the memory usage?
>>
>> Thanks
>> Joe
>>
Re: Memory Usage Problem with Tomcat
Posted by Mike Kienenberger <mk...@gmail.com>.
If you have almost no activity, why not set up a duplicate environment
running the same version of Tomcat and hit your application using
JMeter or some other testing tool?
Or perhaps you can get sent the application http access.log file and
duplicate the exact series of requests that generated the problem in
your dev environment.
That said, Tomcat did often seem to have intrinsic memory issues,
which is another reason I stopped using a few years back.
Also, you might want to ask what the exact stacktrace is. We've had
situations where it was a Tomcat permgen memory issue. See this
article for details -- there are more details in the comments by
others:
http://www.mkyong.com/tomcat/tomcat-javalangoutofmemoryerror-permgen-space/
Again, a disclaimer as I haven't used Tomcat personally in a while,
although some of my colleagues continue to do so for development. And
we don't run app servers using it.
On Tue, Mar 13, 2012 at 2:59 PM, Joe Baldwin <jf...@earthlink.net> wrote:
> OK, I think that I may have run into this before. The ultimate "solution" was to increase memory - however, I am concerned that may have been a quick-fix and not a long-term fix.
>
> The problem is out of memory errors associated with tomcat heap.
>
> I have a webapp (powered primarily by cayenne). The database has *very* little in it. I am essentially serving data (via cayenne & tomcat) and images (via tomcat).
>
> I have a private tomcat instance running on a webhost in a "shared" environment. What this means is that I *absolutely* cannot attach a profiler.
>
> I am being told by the webhost IT people (who are not always accurate in their objectivity) that my app is leaking memory (badly), and that is what caused tomcat to crash.
>
> My intuition tells me that with almost no activity on the website (because it is not live yet) and Cayenne memory management that I should be able to manage memory well, but it is not the case.
>
> So, if my goal is to determine what the problem is, and if I simply increase heap size, won't I just be masking a potential problem? i.e. if the app runs fine for a while, then mysteriously causes tomcat to run out of heap space, then couldn't there be a memory leak?
>
> If there is a memory leak, and I don't see it on my development server, and I *can't* use a profiler on my webhost, then how do I get visibility into the memory usage?
>
> Thanks
> Joe
>