You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Davor Cengija <da...@mail.inet.hr> on 2004/06/16 10:59:12 UTC
Too many open files on Solaris
My application is hitting that 'too many open files' limit on Solaris.
Jun 15, 2004 11:23:01 AM org.apache.tomcat.util.net.PoolTcpEndpoint
acceptSocket
SEVERE: Endpoint ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=8080]
ignored exception: java.net.SocketException: Too many open files
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at
org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:107)
at
org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:387)
at
org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:557)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683)
at java.lang.Thread.run(Thread.java:534)
Jun 15, 2004 11:23:01 AM org.apache.tomcat.util.net.PoolTcpEndpoint
closeServerSocket
SEVERE: Caught exception trying to unlock accept on 8080
java.net.SocketException: Too many open files
Jun 15, 2004 11:23:01 AM org.apache.tomcat.util.net.PoolTcpEndpoint
acceptSocket
WARNING: Reinitializing ServerSocket
I realize that increasing file descriptor limit per process could help a
bit, but my system administrator simply doesn't want to do that (since it
requires restart).
Now, the problem is how to avoid too many open files exception. Any
suggestions regarding Tomcat (or Apache) configuration, coding practice,
some runtime operating system parameters would be greatly appreciated.
Here's my configuration:
Default Solaris 8 (I believe) installation on 2CPU+4GB RAM box. TCP
connection wait timeout (I believe that the correct wording) is decreased
from 240sec to 60sec and that's the only parameter changed on Solaris.
Tomcat 4.1.30 with j2sdk 1.4.2_04 (server mode) with Apache 2.0 as a proxy.
Every single request goes through Apache which then routes it to Tomcat.
600.000 hits and cca 3GB of data daily, with peaks during working hours, of
course, about 50.000 hits per hour. The delivered content consists mostly
of small images (3-20k) delivered directly from a database (servlet sets
content type etc, img src="/preview?id=123" style)
Application lives about 24hours and then dies with the exception shown
above. As far as I can see, all input- and output- streams are flushed and
closed, at least in my application. Cannot guarantee for the underlining
libraries.
Somehow I think that I cannot squeeze the application anymore and that we're
hitting tomcat+solaris limit.
Ahm... whoever managed to completelly read this rather long post I hope will
have some suggestions.
Thanks in advance!
Cheers,
Davor
--
Davor Cengija, dcengija_IQ_Filter_@inet.hr
---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
Re: Too many open files on Solaris
Posted by Davor Cengija <da...@mail.inet.hr>.
QM wrote:
> : > : My application is hitting that 'too many open files' limit on
> : > : Solaris.
> : > [I just left this line in for the archives/searching. Read on]
> [ yet again ;) ]
Yeah baby! :-))
>
> This may be rectified in a positive way, long-term: matching -- or at
> least, semi-matching -- test/prod hardware. One of my clients had such
> a setup. It was beautiful.
It is bad that some customers learn the hard way. They finally accepted to
set up a test machine.
Ahmmm... it is always difficult to fight big companies intertia.
Thanks, QM.
--
Davor Cengija, dcengija_IQ_Filter_@inet.hr
---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
Re: Too many open files on Solaris
Posted by QM <qm...@brandxdev.net>.
: > : My application is hitting that 'too many open files' limit on Solaris.
: > [I just left this line in for the archives/searching. Read on]
[ yet again ;) ]
: Very mission critical. But the problem is that (strange enough) sysadmin
: needs to file a formal restart request (!) in which he needs to explain why
: a restart is needed.
You just explained it. ;)
The suggestions I gave were very quick-hit, yes; because by now 1/ your
problem is more political than technical; and 2/ the technical fixes for
your app are likely long-term.
Some of my clients are very conservative companies that have similar
policies. Telling someone, "this is a huge problem that will eventually
cost you money (in terms of loss of client faith)" tends to light a fire
under the right people.
Unfortunately, you (or someone) will then have to explain to upper mgt
why this didn't bite you till you went to production. I'm not saying
it's your (or anyone's) fault; but someone will have to lay out the
facts and point out a solution.
This may be rectified in a positive way, long-term: matching -- or at
least, semi-matching -- test/prod hardware. One of my clients had such
a setup. It was beautiful.
Good luck.
-QM
--
software -- http://www.brandxdev.net
tech news -- http://www.RoarNetworX.com
---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
Re: Too many open files on Solaris
Posted by Davor Cengija <da...@mail.inet.hr>.
QM wrote:
> On Wed, Jun 16, 2004 at 10:59:12AM +0200, Davor Cengija wrote:
> : My application is hitting that 'too many open files' limit on Solaris.
>
> [I just left this line in for the archives/searching. Read on]
Me too :-)
Ok, this post might sound a little bit funny. Read below.
> : but my system administrator simply doesn't want to do that (since it
> : requires restart).
>
> If this application is mission-critical, your sysadmin may want to
> rethink his/her approach. =)
Very mission critical. But the problem is that (strange enough) sysadmin
needs to file a formal restart request (!) in which he needs to explain why
a restart is needed.
> You could also check out the "ulimit" command.
>
> What sort of load-testing was done on the app before it went live?
It was tested on linux but not on Solaris (long story). Linux performed
quite fine.
<snip>
Thanks for the suggestions. I've already done similar steps you're
suggesting (your post is a good reference now :-).
> How much memory does the container use during peak periods? You could
Java is started with minimum 1GB and maximum 2GB memory footprint. As soon
as Tomcat starts, top says taht memory SIZE is 1.2G and RES about 700MB.
When the application is about to crash, RES is about 1.1GB or slightly
less.
> setup a second Tomcat instance (either on the current host, or a
> different host) and have Apache/mod_jk load-balance between the two.
Hehe, really interesting. My next step is to set up a second Tomcat instance
+ mod_jk on the same box. Another Solaris box is on its way but won't be
available very soon.
Ok, to resume:
QM, your suggestions might be a good quick-fix-how-to for this kind of
problems. I already tried them and still have the problems (although the
application's lifespan is streched from 2hrs to 24hrs for the current
load). My next step is mod_jk balancer.
Ah... my sad story continues :-)
Kind regards,
Davor
--
Davor Cengija, dcengija_IQ_Filter_@inet.hr
---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
Re: Too many open files on Solaris
Posted by QM <qm...@brandxdev.net>.
On Wed, Jun 16, 2004 at 10:59:12AM +0200, Davor Cengija wrote:
: My application is hitting that 'too many open files' limit on Solaris.
[I just left this line in for the archives/searching. Read on]
: but my system administrator simply doesn't want to do that (since it
: requires restart).
If this application is mission-critical, your sysadmin may want to
rethink his/her approach. =)
You could also check out the "ulimit" command.
What sort of load-testing was done on the app before it went live?
: Now, the problem is how to avoid too many open files exception.
The short version: don't open so many files. ;)
The longer version: I'm only half-joking here.
Remember, everything in Unix is (accessible as/represented by) a file,
including network connections. You could limit the number of network
connections in your server.xml <Connector> elements -- or better still,
limit Apache's connections to the container. Reduce the number of DB
pool connections, etc.
You could also check your app for file I/O, such as your code handling
temp files. Slim chance, and it's more of a fine-tuning measure; but it
will help on a political/friendliness level when you can honestly tell
your sysadmin, "I've done all I can do. Your turn."
How much memory does the container use during peak periods? You could
setup a second Tomcat instance (either on the current host, or a
different host) and have Apache/mod_jk load-balance between the two.
-QM
--
software -- http://www.brandxdev.net
tech news -- http://www.RoarNetworX.com
---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
Re: Too many open files on Solaris
Posted by Davor Cengija <da...@mail.inet.hr>.
I've narrowed and hopefully solved the problem described below. At the end,
Solaris+Java 1.4.2+Turbine framework combination is problematic. In short,
Turbine uses resources/ui/skins/default/skin.props (on my system) file for
some configurations etc, and doesn't close it! After a while, that file is
opened 500-600 times and of course causes Too many open files exception
when Tomcat tries to open a socket. The funny thing is that my application
doesn't use the functionalities provided by skin.props; it comes
'configured' by default. Simply removing all references to it solved my Too
many open files problem.
Additionally, the same application on Linux, with the same version of Java
and Tomcat doesn't have that problem. And since I perform my tests on
Linux, it was pretty hard to find the solution.
I hope this post will eventually help somebody.
Cheers,
Davor
Davor Cengija wrote:
> My application is hitting that 'too many open files' limit on Solaris.
>
> Jun 15, 2004 11:23:01 AM org.apache.tomcat.util.net.PoolTcpEndpoint
> acceptSocket
> SEVERE: Endpoint ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=8080]
> ignored exception: java.net.SocketException: Too many open files
> java.net.SocketException: Too many open files
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
> at java.net.ServerSocket.implAccept(ServerSocket.java:448)
> at java.net.ServerSocket.accept(ServerSocket.java:419)
> at
>
org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:107)
> at
>
org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:387)
> at
> org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:557)
> at
>
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683)
> at java.lang.Thread.run(Thread.java:534)
> Jun 15, 2004 11:23:01 AM org.apache.tomcat.util.net.PoolTcpEndpoint
> closeServerSocket
> SEVERE: Caught exception trying to unlock accept on 8080
> java.net.SocketException: Too many open files
> Jun 15, 2004 11:23:01 AM org.apache.tomcat.util.net.PoolTcpEndpoint
> acceptSocket
> WARNING: Reinitializing ServerSocket
>
> I realize that increasing file descriptor limit per process could help a
> bit, but my system administrator simply doesn't want to do that (since it
> requires restart).
>
> Now, the problem is how to avoid too many open files exception. Any
> suggestions regarding Tomcat (or Apache) configuration, coding practice,
> some runtime operating system parameters would be greatly appreciated.
>
> Here's my configuration:
>
> Default Solaris 8 (I believe) installation on 2CPU+4GB RAM box. TCP
> connection wait timeout (I believe that the correct wording) is decreased
> from 240sec to 60sec and that's the only parameter changed on Solaris.
>
> Tomcat 4.1.30 with j2sdk 1.4.2_04 (server mode) with Apache 2.0 as a
> proxy. Every single request goes through Apache which then routes it to
> Tomcat. 600.000 hits and cca 3GB of data daily, with peaks during working
> hours, of course, about 50.000 hits per hour. The delivered content
> consists mostly of small images (3-20k) delivered directly from a database
> (servlet sets content type etc, img src="/preview?id=123" style)
>
> Application lives about 24hours and then dies with the exception shown
> above. As far as I can see, all input- and output- streams are flushed and
> closed, at least in my application. Cannot guarantee for the underlining
> libraries.
>
> Somehow I think that I cannot squeeze the application anymore and that
> we're hitting tomcat+solaris limit.
>
> Ahm... whoever managed to completelly read this rather long post I hope
> will have some suggestions.
>
> Thanks in advance!
>
> Cheers,
> Davor
--
Davor Cengija, dcengija_IQ_Filter_@inet.hr
---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
RE: Too many open files on Solaris
Posted by Davor Cengija <da...@mail.inet.hr>.
Andy Eastham wrote:
> The first thing that came up in Google was this, which might help:
>
>
http://support.bea.com/support_news/product_troubleshooting/Too_Many_Open_Fi
> les_Pattern.html
This is a good reference. Here's a clickable link:
http://tinyurl.com/2acgv
Thanks
--
Davor Cengija, dcengija_IQ_Filter_@inet.hr
---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
RE: Too many open files on Solaris
Posted by Andy Eastham <an...@peakdrive.com>.
The first thing that came up in Google was this, which might help:
http://support.bea.com/support_news/product_troubleshooting/Too_Many_Open_Fi
les_Pattern.html
Andy
> -----Original Message-----
> From: news [mailto:news@sea.gmane.org] On Behalf Of Davor Cengija
> Sent: 16 June 2004 09:59
> To: tomcat-user@jakarta.apache.org
> Subject: Too many open files on Solaris
>
> My application is hitting that 'too many open files' limit on Solaris.
>
> Jun 15, 2004 11:23:01 AM org.apache.tomcat.util.net.PoolTcpEndpoint
> acceptSocket
> SEVERE: Endpoint ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=8080]
> ignored exception: java.net.SocketException: Too many open files
> java.net.SocketException: Too many open files
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
> at java.net.ServerSocket.implAccept(ServerSocket.java:448)
> at java.net.ServerSocket.accept(ServerSocket.java:419)
> at
> org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(Default
> ServerSocketFactory.java:107)
> at
> org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.ja
> va:387)
> at
> org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:557)
> at
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.j
> ava:683)
> at java.lang.Thread.run(Thread.java:534)
> Jun 15, 2004 11:23:01 AM org.apache.tomcat.util.net.PoolTcpEndpoint
> closeServerSocket
> SEVERE: Caught exception trying to unlock accept on 8080
> java.net.SocketException: Too many open files
> Jun 15, 2004 11:23:01 AM org.apache.tomcat.util.net.PoolTcpEndpoint
> acceptSocket
> WARNING: Reinitializing ServerSocket
>
> I realize that increasing file descriptor limit per process could help a
> bit, but my system administrator simply doesn't want to do that (since it
> requires restart).
>
> Now, the problem is how to avoid too many open files exception. Any
> suggestions regarding Tomcat (or Apache) configuration, coding practice,
> some runtime operating system parameters would be greatly appreciated.
>
> Here's my configuration:
>
> Default Solaris 8 (I believe) installation on 2CPU+4GB RAM box. TCP
> connection wait timeout (I believe that the correct wording) is decreased
> from 240sec to 60sec and that's the only parameter changed on Solaris.
>
> Tomcat 4.1.30 with j2sdk 1.4.2_04 (server mode) with Apache 2.0 as a
> proxy.
> Every single request goes through Apache which then routes it to Tomcat.
> 600.000 hits and cca 3GB of data daily, with peaks during working hours,
> of
> course, about 50.000 hits per hour. The delivered content consists mostly
> of small images (3-20k) delivered directly from a database (servlet sets
> content type etc, img src="/preview?id=123" style)
>
> Application lives about 24hours and then dies with the exception shown
> above. As far as I can see, all input- and output- streams are flushed and
> closed, at least in my application. Cannot guarantee for the underlining
> libraries.
>
> Somehow I think that I cannot squeeze the application anymore and that
> we're
> hitting tomcat+solaris limit.
>
> Ahm... whoever managed to completelly read this rather long post I hope
> will
> have some suggestions.
>
> Thanks in advance!
>
> Cheers,
> Davor
> --
> Davor Cengija, dcengija_IQ_Filter_@inet.hr
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org