You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Matthew Boeckman <ma...@saepio.com> on 2003/11/24 21:59:16 UTC

open filehandles

Hello list,

	I have a problem that has occurred under tomcat-4.1.12/j2sdk1.4.1_02 
that caused me to want to ask the group a question. The problem was "Too 
many open files", java.io.FileNotFoundException . In looking into this, 
I notice that Tomcat seems to keep a LOT of open file handles around. 
Let me explain our setup, we run multiple (~12-15) virtual hosts from 
the same tomcat engine, with apache in front handing off via mod_jk, all 
on Linux 2.4.9.

	What I noticed is that Tomcat seems to keep a file handle to every .jar 
file in it's own CLASSPATH (/usr/local/tomcat-4.1.12/shared /commons 
/lib, etc) as well as in the unique hosts' classpath, per unique thread. 
So what I'm seeing is 5000+ file handles open to the same .jar files 
over and over again.

My questions are twofold: Is this tunable/fixable/changeable? Why are 
.jar file handles maintained in an open state indefinately, when 
classfiles/config files etc are not?

Any replies appreciated

-Matthew

-- 
Matthew Boeckman			(816) 777-2160
Manager - Systems Integration		Saepio Technologies


---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org


Re: open filehandles

Posted by Matthew Boeckman <ma...@saepio.com>.
revilak@g88.net wrote:
>>>As a shorter-term workaround, you might try one or more of the
>>>following:
>>>
>>> - if open filehandles are not maintained for individual class files,
>>>   try expanding your jars into each classloader's corresponding
>>>   "classes" directory (i.e shared/lib/*.jar into shared/classes,
>>>   etc).
>>
> 
>>I'm confused by this, why would I _want_ filehandles maintained for
>>individual classes?
> 
> 
> You don't.
> 
> 
> 
>>Or am I misunderstanding your suggestion? Right now tomcat touches a
>>class and moves on, which is ideally what I want it to do with the
>>jar's.
> 
> 
> Or, perhaps I misunderstood your observations.  Let me try to break it
> down into a little more detail.
> 
>  - single class files: the class loader opens the .class file, loads the
>    class than closes the file.  No lingering open file handles.
>    (My read through your initial message gave me the impression that
>    this was the behavior you were observing).

that is correct

> 
>  - jar files: classloader opens the jar file, does
>    decompression if necessary, and memory mapping of the jar contents,
>    then loads some number of classes from the jar (but not necessarily
>    all of them).  From an efficiency standpoint, it kind of makes
>    sense to keep a handle open, so that it's not necessary to do all
>    of that work repeatedly when searching for a class (see also
>    http://java.sun.com/webservices/faq.html#size)
> 
> If open FD's to jar files are the problem, try working around it by
> replacing the jars with the equivalent directory tree of expanded
> class files (where each class file will be opened, read, and closed as
> it is needed).
> 

and that makes perfect sense now. I just needed it spelled out a little 
more I guess. Having read that FAQ the 'keeping those file handles open' 
does make sense to me.

What we have seen is our tomcat engine crash twice in the last 72 hours, 
and hadn't seen enough info to track down the specific cause, so I was 
really trying to just trim file handles down, and the .jar's seemed a 
likely candidate. However since my last post I have discovered that this 
looks like a nasty code bug in our software more than a simple 
tomcat-is-a-filehog problem, so we'll hit it from there. FWIW I did 
uncompress and recombine all jars in tomcat and my app into 3 .jar files 
and saw some significant decreases (1200+) in the number of filehandles 
open. So again, many thanks!

-matthew

-- 
Matthew Boeckman			(816) 777-2160
Manager - Systems Integration		Saepio Technologies


---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org


Re: open filehandles

Posted by sr...@g88.net.
>> As a shorter-term workaround, you might try one or more of the
>> following:
>>
>>  - if open filehandles are not maintained for individual class files,
>>    try expanding your jars into each classloader's corresponding
>>    "classes" directory (i.e shared/lib/*.jar into shared/classes,
>>    etc).

> I'm confused by this, why would I _want_ filehandles maintained for
> individual classes?

You don't.


> Or am I misunderstanding your suggestion? Right now tomcat touches a
> class and moves on, which is ideally what I want it to do with the
> jar's.

Or, perhaps I misunderstood your observations.  Let me try to break it
down into a little more detail.

 - single class files: the class loader opens the .class file, loads the
   class than closes the file.  No lingering open file handles.
   (My read through your initial message gave me the impression that
   this was the behavior you were observing).

 - jar files: classloader opens the jar file, does
   decompression if necessary, and memory mapping of the jar contents,
   then loads some number of classes from the jar (but not necessarily
   all of them).  From an efficiency standpoint, it kind of makes
   sense to keep a handle open, so that it's not necessary to do all
   of that work repeatedly when searching for a class (see also
   http://java.sun.com/webservices/faq.html#size)

If open FD's to jar files are the problem, try working around it by
replacing the jars with the equivalent directory tree of expanded
class files (where each class file will be opened, read, and closed as
it is needed).

Sorry if my earlier message wasn't clear.

-- 
Steve

---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org


Re: open filehandles

Posted by Matthew Boeckman <ma...@saepio.com>.
srevilak@g88.net wrote:
> mboeckman> I have a problem that has occurred under
> mboeckman> tomcat-4.1.12/j2sdk1.4.1_02 that caused me to want to ask
> mboeckman> the group a question. The problem was "Too many open
> mboeckman> files", java.io.FileNotFoundException .
> 
> mboeckman> What I noticed is that Tomcat seems to keep a file handle
> mboeckman> to every .jar file in it's own CLASSPATH
> mboeckman> (/usr/local/tomcat-4.1.12/shared /commons /lib, etc) as
> mboeckman> well as in the unique hosts' classpath, per unique
> mboeckman> thread. So what I'm seeing is 5000+ file handles open to
> mboeckman> the same .jar files over and over again.
> 
> mboeckman> Why are .jar file handles
> mboeckman> maintained in an open state indefinately, when
> mboeckman> classfiles/config files etc are not?
> 
> jfarcand> You need to update to Tomcat 5 if you want that problem
> jfarcand> fixed. The required changes are too critical to apply them
> jfarcand> to the 4.1.x trunk. Wait a couple of days and try it with
> jfarcand> "soon to be release" 5.0.15, which is a stable Tomcat 5.
> 
> Ouch. :(

Thank you for the response. I was not looking forward to deploying a 
brand new tomcat in my production environment. I have a few additional 
questions/comments inline:

> 
> As a shorter-term workaround, you might try one or more of the
> following:
> 
>  - if open filehandles are not maintained for individual class files,
>    try expanding your jars into each classloader's corresponding
>    "classes" directory (i.e shared/lib/*.jar into shared/classes,
>    etc).

I'm confused by this, why would I _want_ filehandles maintained for 
individual classes? Or am I misunderstanding your suggestion? Right now 
tomcat touches a class and moves on, which is ideally what I want it to 
do with the jar's.

> 
>  - try combining multiple jars into a single jar: make a temporary
>    directory, unpack each jar into it, then jar up the temporary
>    directory.  Again, you'd need to do this on a per-classloader
>    basis
> 
> If doing this, be careful of signed jars.  You'll need to unsign them
> by removing any META-INF/*.SF, META-INF/*.RSA, and META-INF/MANIFEST.MF
> files.

I was playing around with this and achieved a decent amount of success. 
Can the same be done with the tomcat supplied jar's (say /server/lib/*) 
without negative effect?

> 
>  - try increasing the number of file descriptors at the os level.
>    With a solaris machine, that's rlim_fd_cur and rlim_fd_max in
>    /etc/system.  I'm not sure about linux (/proc/sys/fs/file-max?)

This is the last thing I'm looking into. I'd much rather get the 
application to behave than to tune a kernel for a poorly performing app, 
but if nothing else works I'll finish with that.

Many thanks,

Matthew

-- 
Matthew Boeckman			(816) 777-2160
Manager - Systems Integration		Saepio Technologies


---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org


Re: open filehandles

Posted by sr...@g88.net.
mboeckman> I have a problem that has occurred under
mboeckman> tomcat-4.1.12/j2sdk1.4.1_02 that caused me to want to ask
mboeckman> the group a question. The problem was "Too many open
mboeckman> files", java.io.FileNotFoundException .

mboeckman> What I noticed is that Tomcat seems to keep a file handle
mboeckman> to every .jar file in it's own CLASSPATH
mboeckman> (/usr/local/tomcat-4.1.12/shared /commons /lib, etc) as
mboeckman> well as in the unique hosts' classpath, per unique
mboeckman> thread. So what I'm seeing is 5000+ file handles open to
mboeckman> the same .jar files over and over again.

mboeckman> Why are .jar file handles
mboeckman> maintained in an open state indefinately, when
mboeckman> classfiles/config files etc are not?

jfarcand> You need to update to Tomcat 5 if you want that problem
jfarcand> fixed. The required changes are too critical to apply them
jfarcand> to the 4.1.x trunk. Wait a couple of days and try it with
jfarcand> "soon to be release" 5.0.15, which is a stable Tomcat 5.

Ouch. :(

As a shorter-term workaround, you might try one or more of the
following:

 - if open filehandles are not maintained for individual class files,
   try expanding your jars into each classloader's corresponding
   "classes" directory (i.e shared/lib/*.jar into shared/classes,
   etc).

 - try combining multiple jars into a single jar: make a temporary
   directory, unpack each jar into it, then jar up the temporary
   directory.  Again, you'd need to do this on a per-classloader
   basis.

If doing this, be careful of signed jars.  You'll need to unsign them
by removing any META-INF/*.SF, META-INF/*.RSA, and META-INF/MANIFEST.MF
files.

 - try increasing the number of file descriptors at the os level.
   With a solaris machine, that's rlim_fd_cur and rlim_fd_max in
   /etc/system.  I'm not sure about linux (/proc/sys/fs/file-max?)

-- 
Steve

---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org


Re: open filehandles

Posted by Jean-Francois Arcand <jf...@apache.org>.
You need to update to Tomcat 5 if you want that problem fixed. The 
required changes are too critical to apply them to the 4.1.x trunk. Wait 
a couple of days and try it with "soon to be release" 5.0.15, which is a 
stable Tomcat 5.

-- Jeanfrancois

Matthew Boeckman wrote:

> Hello list,
>
>     I have a problem that has occurred under 
> tomcat-4.1.12/j2sdk1.4.1_02 that caused me to want to ask the group a 
> question. The problem was "Too many open files", 
> java.io.FileNotFoundException . In looking into this, I notice that 
> Tomcat seems to keep a LOT of open file handles around. Let me explain 
> our setup, we run multiple (~12-15) virtual hosts from the same tomcat 
> engine, with apache in front handing off via mod_jk, all on Linux 2.4.9.
>
>     What I noticed is that Tomcat seems to keep a file handle to every 
> .jar file in it's own CLASSPATH (/usr/local/tomcat-4.1.12/shared 
> /commons /lib, etc) as well as in the unique hosts' classpath, per 
> unique thread. So what I'm seeing is 5000+ file handles open to the 
> same .jar files over and over again.
>
> My questions are twofold: Is this tunable/fixable/changeable? Why are 
> .jar file handles maintained in an open state indefinately, when 
> classfiles/config files etc are not?
>
> Any replies appreciated
>
> -Matthew
>


---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org