You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@struts.apache.org by Bryan Taylor <br...@yahoo.com> on 2008/02/29 20:49:03 UTC
Severe production issue: struts2-core-2.0.9.jar has hundres of file handles
We're troubleshooting a severe problem in our production system. We get errors like this in tomcat:
SEVERE: Socket accept failed
org.apache.tomcat.jni.Error: Too many open files
We're on linux and run lsof to show the files tomcat has open, which shows hundreds (763 in one example) of distinct open files for struts2-core-2.0.9.jar . The large number of these causes us to hit the limit on the number of open files (1024 by default).
In a development setting, I run the following script to display open file handles to this jar every second:
[root]# while true; do lsof -p $TOMCAT_PID | grep struts2-core-2.0.9.jar | wc -l; sleep 1; done
When I go to our login page (a freemarker template served by a struts action) if I hit refresh on the page as quickly as I can, I see the numbers start at 2 or 3 and then go up in jumps of 20 at a time (eg 23, 43, 63, 83). After a while, say 30 seconds, the number will drop again, perhaps due to garbage collection. It appears that under our production load, the increases are beating the decreases until we hit the open file limit.
We have attempted to duplicate this with the struts showcase sample app. One user by simply hitting refresh a lot can make the file handle count go into the 30's.
Any ideas on what could be happening? Any help is greatly appreciate -- this is causing us big problems.
____________________________________________________________________________________
Looking for last minute shopping deals?
Find them fast with Yahoo! Search. http://tools.search.yahoo.com/newsearch/category.php?category=shopping
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@struts.apache.org
For additional commands, e-mail: dev-help@struts.apache.org
Re: Severe production issue: struts2-core-2.0.9.jar has hundres of
file handles
Posted by "Hilbert Schraal, Hippe IT" <hi...@hippe-it.nl>.
Same problem happens in 2.0.11. We have fixed this particular problem by
creating the freemarker.properties as described on the performance tuning
page (http://struts.apache.org/2.0.11/docs/performance-tuning.html)
Note that we had a similar problem with *-conversion.properties files. If
you encounter this problem, set:
struts.devMode=false
struts.configuration.xml.reload=false
Hilbert
Bryan Taylor-2 wrote:
>
> Two things catch my eye in the second document:
> - Freemarker checks every 500ms by default for template changes
> - The struts.freemarker.templatesCache property added in Struts 2.0.10
>
> I did a little debugging throught the struts code to find the spot where
> the file handle count goes up. If I start in the FreemarkerResult class, I
> see a jump from 2 file handles to 22 file handles when the call to
> template.process(model, getWriter()) happens. I traced into this to
> understand it a little better. It first goes through some Freemarker code,
> and then comes back into the struts2 code for the various tags. My
> template has a few different tags, so I focused on the form tag.
>
> In org.apache.struts2.views.freemarker.tags.CallbackWriter.onStart() we
> call bean.start(this), where bean is an instance of
> org.apache.struts2.components.Form. The code is invoked is from Form's
> parent, org.apache.struts2.components.ClosingUIBean.start(Writer) which
> calls org.apache.struts2.components.UIBean.mergeTemplate(Writer,
> Template), from its parent class. This creates the template engine and
> context and calls
> org.apache.struts2.components.template.FreemarkerTemplateEngine.renderTemplate(TemplateRenderingContext)
>
> In that class we eventually call template.process(model, writer) which
> goes into the Freemarker code. When I do this, I see the file handle count
> go from 2 to 6. This is consistent with the file handle leak coming at the
> rate of 1 handle per freemarker template, because xhtml/form.ftl includes
> three other subtemplates:
> xhtml/form-validate.ftl
> simple/form.ftl
> xhtml/control.ftl
> I'm guess the other file handles come from the other tags in my page.
>
> Given the two observations above from the second tuning document, keeping
> the jar file handle open seems like a likely consequence.
>
> ----- Original Message ----
> From: Jeromy Evans <je...@blueskyminds.com.au>
> To: Struts Developers List <de...@struts.apache.org>
> Sent: Friday, February 29, 2008 11:25:46 PM
> Subject: Re: Severe production issue: struts2-core-2.0.9.jar has hundres
> of file handles
>
> Hi Brian,
>
> Yes, you're right that by default the templates are retrieved
> from the
> jar. It's a normal performance improvement to extract them from
> the
> java to improve performance and this may be why others haven't
> noticed
> the problem.
>
> Do you have time to copy them out and observe the difference?
>
> See Section 1 of this document on info about extracting the
> templates:
> http://cwiki.apache.org/confluence/display/S2WIKI/Creating+a+custom+Dojo+profile+for+Struts+2.0.x
>
> And here: http://struts.apache.org/2.0.11/docs/performance-tuning.html
>
> Bryan Taylor wrote:
>> I'm thinking more about this. It's very interesting that the
>> open file is the struts2-core jar.
>> This is obviously opened by a class loader to get class
>> definitions and resource files.
>>
>> Several questions come to mind:
>> - What files are being retrieving?
>> - What classloader is doing this? Is it the normal Tomcat
>> webapp classloader?
>> - What would cause a classloader to fail to release a file
>> handle?
>>
>> Perhaps we are retrieving the freemarker templates for something
>> like form.ftl from the
>> jar?
>>
>> ----- Original Message ----
>> From: Dave Newton <ne...@yahoo.com>
>> To: Struts Developers List <de...@struts.apache.org>
>> Sent: Friday, February 29, 2008 2:06:59 PM
>> Subject: Re: Severe production issue: struts2-core-2.0.9.jar has
>> hundres of file handles
>>
>> FWIW (not much), I can't duplicate this with S2.0.11 under Mac
>> OS X; I may
>> have a report from both Windows and Linux boxen in a little
>> bit.
>>
>> Dave
>>
>> --- Bryan Taylor <br...@yahoo.com> wrote:
>>
>>
>>> We're troubleshooting a severe problem in our production
>>> system. We get
>>> errors like this in tomcat:
>>> SEVERE: Socket accept failed
>>> org.apache.tomcat.jni.Error: Too many open files
>>>
>>> We're on linux and run lsof to show the files tomcat has
>>> open, which shows
>>> hundreds (763 in one example) of distinct open files for
>>> struts2-core-2.0.9.jar . The large number of these causes us
>>> to hit the
>>> limit on the number of open files (1024 by default).
>>>
>>> In a development setting, I run the following script to
>>> display open file
>>> handles to this jar every second:
>>> [root]# while true; do lsof -p $TOMCAT_PID | grep
>>> struts2-core-2.0.9.jar |
>>> wc -l; sleep 1; done
>>>
>>> When I go to our login page (a freemarker template served by
>>> a struts
>>> action) if I hit refresh on the page as quickly as I can,
>>> I see the numbers
>>> start at 2 or 3 and then go up in jumps of 20 at a time
>>> (eg 23, 43, 63,
>>> 83). After a while, say 30 seconds, the number will drop
>>> again, perhaps due
>>> to garbage collection. It appears that under our production
>>> load, the
>>> increases are beating the decreases until we hit the open
>>> file limit.
>>>
>>> We have attempted to duplicate this with the struts showcase
>>> sample app.
>>> One user by simply hitting refresh a lot can make the file
>>> handle count go
>>> into the 30's.
>>>
>>> Any ideas on what could be happening? Any help is greatly
>>> appreciate --
>>> this is causing us big problems.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>> ____________________________________________________________________________________
>>
>>> Looking for last minute shopping deals?
>>> Find them fast with Yahoo! Search.
>>> http://tools.search.yahoo.com/newsearch/category.php?category=shopping
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@struts.apache.org
>>> For additional commands, e-mail: dev-help@struts.apache.org
>>>
>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@struts.apache.org
>> For additional commands, e-mail: dev-help@struts.apache.org
>>
>>
>>
>>
>>
>>
>>
>> ____________________________________________________________________________________
>> Be a better friend, newshound, and
>> know-it-all with Yahoo! Mobile. Try it now.
>> http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@struts.apache.org
>> For additional commands, e-mail: dev-help@struts.apache.org
>>
>>
>>
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@struts.apache.org
> For additional commands, e-mail: dev-help@struts.apache.org
>
>
>
>
>
>
>
>
> ____________________________________________________________________________________
> Looking for last minute shopping deals?
> Find them fast with Yahoo! Search.
> http://tools.search.yahoo.com/newsearch/category.php?category=shopping
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@struts.apache.org
> For additional commands, e-mail: dev-help@struts.apache.org
>
>
>
--
View this message in context: http://www.nabble.com/Severe-production-issue%3A-struts2-core-2.0.9.jar-has-hundres-of-file-handles-tp15766227p15876779.html
Sent from the Struts - Dev mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@struts.apache.org
For additional commands, e-mail: dev-help@struts.apache.org
Re: Severe production issue: struts2-core-2.0.9.jar has hundres of file handles
Posted by Dave Newton <ne...@yahoo.com>.
FWIW (not much), I can't duplicate this with S2.0.11 under Mac OS X; I may
have a report from both Windows and Linux boxen in a little bit.
Dave
--- Bryan Taylor <br...@yahoo.com> wrote:
> We're troubleshooting a severe problem in our production system. We get
> errors like this in tomcat:
> SEVERE: Socket accept failed
> org.apache.tomcat.jni.Error: Too many open files
>
> We're on linux and run lsof to show the files tomcat has open, which shows
> hundreds (763 in one example) of distinct open files for
> struts2-core-2.0.9.jar . The large number of these causes us to hit the
> limit on the number of open files (1024 by default).
>
> In a development setting, I run the following script to display open file
> handles to this jar every second:
> [root]# while true; do lsof -p $TOMCAT_PID | grep struts2-core-2.0.9.jar |
> wc -l; sleep 1; done
>
> When I go to our login page (a freemarker template served by a struts
> action) if I hit refresh on the page as quickly as I can, I see the numbers
> start at 2 or 3 and then go up in jumps of 20 at a time (eg 23, 43, 63,
> 83). After a while, say 30 seconds, the number will drop again, perhaps due
> to garbage collection. It appears that under our production load, the
> increases are beating the decreases until we hit the open file limit.
>
> We have attempted to duplicate this with the struts showcase sample app.
> One user by simply hitting refresh a lot can make the file handle count go
> into the 30's.
>
> Any ideas on what could be happening? Any help is greatly appreciate --
> this is causing us big problems.
>
>
>
>
>
>
>
>
____________________________________________________________________________________
> Looking for last minute shopping deals?
> Find them fast with Yahoo! Search.
> http://tools.search.yahoo.com/newsearch/category.php?category=shopping
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@struts.apache.org
> For additional commands, e-mail: dev-help@struts.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@struts.apache.org
For additional commands, e-mail: dev-help@struts.apache.org