You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Mladen Adamovic <ad...@blic.net> on 2006/09/23 18:37:46 UTC
static objects, server.xml and preventing running out of available
file descriptors
Hi all!
I have many problems recently with java.io.IOException: Too many open
files on my Tomcat server for /Online Utility/
<http://www.online-utility.org> (http://www.online-utility.org) web site.
The server runs Linux (Suse 9.3). My web server stop running when runs
out of available file descriptors.
I know it is a common problem (at least it is not uncommon), I googled a
lot about that problems and find a lot, but still I have some questions
and issues.
I want to do everything I can to prevent that behavior.
After analyzing "lsof" (see man lsof) I have seen a lot of files open by
WordNetProcessor class I use.
I use WordNetProcessor in many JSP files and it seems if I use it in
static way using
<%! static WordNetProcessor wordNetProcessor=new WordNetProcessor(); %>
it won't lock many files.
*QUESTIONS:*
*** So, could I assume it is favorably to use static objects whenever
possible?
*** Could I also tweak my server.xml somehow to use less threads and
therefore to lock less files?
Currently I have :
<Connector port="80" maxHttpHeaderSize="8192"
maxThreads="150" minSpareThreads="25" maxSpareThreads="75"
enableLookups="false" redirectPort="8443" acceptCount="100"
connectionTimeout="20000" disableUploadTimeout="true" />
Maybe I should lower to maxThreads="20" minSpareThreads="15"
maxSpareThreads="15"? Or something?
*** Should I check places where I use BufferedReader for something? If
yes, for what?
*** Should I place some System.gc() at few places in code?
*** What else should I do?
BTW, Since I use Virtuozzo powered VPS for its hosting, it seems that
Virtuozzo lower the number of available file descriptors since most
likely they are shared among all virtual private servers.
Thanks in advance for any advice / comment.
--
Mladen Adamovic
http://www.online-utility.org
http://www.cheapvps.info
http://www.vpsreview.com
Re: static objects, server.xml and preventing running out of available
file descriptors
Posted by Mladen Adamovic <ad...@blic.net>.
Darryl Miles wrote:
>> Also, I've put in /etc/profile.local the line "ulimit -n 8192",
>> hoping it will help.
>
> WARNING: If you are unix the JVM uses the select() then increasing the
> ulimit above the default 1024 maybe dangerous.
Is it dangerous in Linux also?
--
Mladen Adamovic
http://www.online-utility.org
http://www.cheapvps.info
http://www.vpsreview.com
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: static objects, server.xml and preventing running out of available
file descriptors
Posted by Darryl Miles <da...@netbauds.net>.
Mladen Adamovic wrote:
> Also, I've put in /etc/profile.local the line "ulimit -n 8192", hoping
> it will help.
WARNING: If you are unix the JVM uses the select() then increasing the
ulimit above the default 1024 maybe dangerous. This is because the
default select() usage set only allocates enough memory for 1024
descriptors in its bitmap. This can result in the bitmap modification
functions FD_SET() from scribbling over memory off the end of the memory
allocation.
It is also necessary to audit all system libraries that the JVM loads
and uses during the course of running, everything that waits for
non-blocking IO must be audited.
If you are unix take a look at "ls -l /proc/$pid/fd/" to see where all
your file descriptors are going.
Darryl
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: static objects, server.xml and preventing running out of available
file descriptors
Posted by Mladen Adamovic <ad...@blic.net>.
I've put System.gc() in one place in the code and I'm seeing that "lsof
| wc -l" shows that my web application uses less file descriptors as
time go on :)
I hope the server won't run out of available file descriptors in the future.
Also, I've put in /etc/profile.local the line "ulimit -n 8192", hoping
it will help.
I'm planing to make a cron job which will check server each 10 minutes
and reboot the computer if web server is unavailable at the third time
(wget returns error).
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: static objects, server.xml and preventing running out of available file descriptors
Posted by Leon Rosenberg <ro...@googlemail.com>.
Hi Mladen,
On 9/24/06, Mladen Adamovic <ad...@blic.net> wrote:
> Leon Rosenberg wrote:
> >> I use WordNetProcessor in many JSP files and it seems if I use it in
> >> static way using
> >> <%! static WordNetProcessor wordNetProcessor=new WordNetProcessor(); %>
> >> it won't lock many files.
> >
> > then your WordNetProcessor class is probably buggy. Maybe you should
> > show us your code to detect where the problem is.
>
> That class uses other library (JWNL) which opens some files and I cannot
> modify that other library and I think it is not practical at this moment
> to search for bugs in that library.
No, but maybe you are using it the wrong way?
>
>
> >
> > you can lower the thread count, but I think it won't help you much. Do
> > you know how many concurrent users your site actually have?
>
> Currently, it don't have many concurrent users (~400 uniques a day) but
> my traffic goes up, since website is just 6 months old.
> Also, some users tends to send many requests a day. Some even 500
> requests a day (seems to be manual, from log files).
> At this moment 5 threads would be probably enough, but in the future I
> might need significantly more in case of i.e. slashdot effect or DoS
> attacks.
Still, decreasing threads isn't a solution :-) It may work for some
months, but in long term you should try to find out who's eating away
your file descriptors, or change the hoster.
Btw, I like your site, especially the SEO tools :-)
Leon
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: static objects, server.xml and preventing running out of available
file descriptors
Posted by Mladen Adamovic <ad...@blic.net>.
Leon Rosenberg wrote:
>> I use WordNetProcessor in many JSP files and it seems if I use it in
>> static way using
>> <%! static WordNetProcessor wordNetProcessor=new WordNetProcessor(); %>
>> it won't lock many files.
>
> then your WordNetProcessor class is probably buggy. Maybe you should
> show us your code to detect where the problem is.
That class uses other library (JWNL) which opens some files and I cannot
modify that other library and I think it is not practical at this moment
to search for bugs in that library.
>
> you can lower the thread count, but I think it won't help you much. Do
> you know how many concurrent users your site actually have?
Currently, it don't have many concurrent users (~400 uniques a day) but
my traffic goes up, since website is just 6 months old.
Also, some users tends to send many requests a day. Some even 500
requests a day (seems to be manual, from log files).
At this moment 5 threads would be probably enough, but in the future I
might need significantly more in case of i.e. slashdot effect or DoS
attacks.
> regards
> Leon
Thanks for your answer.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: static objects, server.xml and preventing running out of available file descriptors
Posted by Leon Rosenberg <ro...@googlemail.com>.
thats probably true (depends on what lies under the
bufferedreader/stream, you probably mean a file?)
But its generally extremely bad taste to rely on gc when you can fix
it by adding a three-liner.
that would also expain why your open files become closed after
System.gc() is called.
However, you dont know when the gc will collect something, so better
add the close call.
regards
leon
On 9/25/06, Mladen Adamovic <ad...@blic.net> wrote:
> Martin Gainty wrote:
> > I think you may have what Leon called a 'resource allocation issue' that may not be corrected until you actually look at the code
> > to determine who or what is grabbing all of the file handles (and not closing them)
> >
>
> Yeah, but if one uses
> BufferedReader br=new BufferedReader(...);
> and doesn't do br.close()
> that file descriptor will be automatically closed on garbage collection,
> when the object isn't in use any longer, right?
>
>
> --
> Mladen Adamovic
> http://www.online-utility.org
> http://www.cheapvps.info
> http://www.vpsreview.com
>
>
>
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: static objects, server.xml and preventing running out of available
file descriptors
Posted by Mladen Adamovic <ad...@blic.net>.
Martin Gainty wrote:
> I think you may have what Leon called a 'resource allocation issue' that may not be corrected until you actually look at the code
> to determine who or what is grabbing all of the file handles (and not closing them)
>
Yeah, but if one uses
BufferedReader br=new BufferedReader(...);
and doesn't do br.close()
that file descriptor will be automatically closed on garbage collection,
when the object isn't in use any longer, right?
--
Mladen Adamovic
http://www.online-utility.org
http://www.cheapvps.info
http://www.vpsreview.com
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: static objects, server.xml and preventing running out of available file descriptors
Posted by Martin Gainty <mg...@hotmail.com>.
Having worked in environments where servers get rebooted at odd times for unknown reasons I'll ask the dumb question
What happens to the other processes that are running when the server is rebooted?
What happens to the logs?
I think you may have what Leon called a 'resource allocation issue' that may not be corrected until you actually look at the code
to determine who or what is grabbing all of the file handles (and not closing them)
M-
*********************************************************************
This email message and any files transmitted with it contain confidential
information intended only for the person(s) to whom this email message is
addressed. If you have received this email message in error, please notify
the sender immediately by telephone or email and destroy the original
message without making a copy. Thank you.
----- Original Message -----
From: "Mladen Adamovic" <ad...@blic.net>
To: "Tomcat Users List" <us...@tomcat.apache.org>
Sent: Monday, September 25, 2006 4:51 AM
Subject: Re: static objects, server.xml and preventing running out of available file descriptors
>I prepared one bash script which aim is to reboot the computer if its
> web server is down for approx. 30 minutes.
> I think that bash script might be useful for others on this list, so I'm
> posting it here.
> But, I'm not sure what is the best way to run that script.
> I'm using Suse 9.3 on the server. Should I make it runnable in
> /etc/init.d as daemon or there is some easier way?
>
> Script's code:
>
> #!/bin/bash
> #bash script which reboots the computer when web server is down for
> approx. 30 minutes
> serverFailure=0
> while [ true ]
> do
> wget -t 1 -T 30 -q www.mysite.com
>
> if [ $? != 0 ]; then
> #echo 'server is down!';
> serverFailure=$(($serverFailure + 1));
> if [ $serverFailure -gt 3 ]; then
> echo 'too many server failures... going to reboot' >>
> ~/reboot.log ;
> date >> ~/reboot.log
> reboot
>
> fi
> else
> echo 'server is up!';
> serverFailure=0
> fi
> #echo 'ServerFailure='$serverFailure;
> sleep 7m
> done
>
>
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: static objects, server.xml and preventing running out of available
file descriptors
Posted by Mladen Adamovic <ad...@blic.net>.
I prepared one bash script which aim is to reboot the computer if its
web server is down for approx. 30 minutes.
I think that bash script might be useful for others on this list, so I'm
posting it here.
But, I'm not sure what is the best way to run that script.
I'm using Suse 9.3 on the server. Should I make it runnable in
/etc/init.d as daemon or there is some easier way?
Script's code:
#!/bin/bash
#bash script which reboots the computer when web server is down for
approx. 30 minutes
serverFailure=0
while [ true ]
do
wget -t 1 -T 30 -q www.mysite.com
if [ $? != 0 ]; then
#echo 'server is down!';
serverFailure=$(($serverFailure + 1));
if [ $serverFailure -gt 3 ]; then
echo 'too many server failures... going to reboot' >>
~/reboot.log ;
date >> ~/reboot.log
reboot
fi
else
echo 'server is up!';
serverFailure=0
fi
#echo 'ServerFailure='$serverFailure;
sleep 7m
done
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: static objects, server.xml and preventing running out of available file descriptors
Posted by Leon Rosenberg <ro...@googlemail.com>.
On 9/23/06, Mladen Adamovic <ad...@blic.net> wrote:
> Hi all!
Hi
>
> I have many problems recently with java.io.IOException: Too many open
> files on my Tomcat server for /Online Utility/
> <http://www.online-utility.org> (http://www.online-utility.org) web site.
> The server runs Linux (Suse 9.3). My web server stop running when runs
> out of available file descriptors.
>
> I know it is a common problem (at least it is not uncommon), I googled a
> lot about that problems and find a lot, but still I have some questions
> and issues.
> I want to do everything I can to prevent that behavior.
how about setting the ulimit higher?
ulimit -n 4096
for example. default would be 1024 i suppose.
>
> After analyzing "lsof" (see man lsof) I have seen a lot of files open by
> WordNetProcessor class I use.
> I use WordNetProcessor in many JSP files and it seems if I use it in
> static way using
>
> <%! static WordNetProcessor wordNetProcessor=new WordNetProcessor(); %>
>
> it won't lock many files.
then your WordNetProcessor class is probably buggy. Maybe you should
show us your code to detect where the problem is.
>
> *QUESTIONS:*
> *** So, could I assume it is favorably to use static objects whenever
> possible?
No! It's very use-case dependent and there is probably no general
answer to this question, since it can differ from application to
application.
>
> *** Could I also tweak my server.xml somehow to use less threads and
> therefore to lock less files?
> Currently I have :
> <Connector port="80" maxHttpHeaderSize="8192"
> maxThreads="150" minSpareThreads="25" maxSpareThreads="75"
> enableLookups="false" redirectPort="8443" acceptCount="100"
> connectionTimeout="20000" disableUploadTimeout="true" />
> Maybe I should lower to maxThreads="20" minSpareThreads="15"
> maxSpareThreads="15"? Or something?
you can lower the thread count, but I think it won't help you much. Do
you know how many concurrent users your site actually have?
>
>
> *** Should I check places where I use BufferedReader for something? If
> yes, for what?
I think you should check whether you close the files properly after reading.
>
> *** Should I place some System.gc() at few places in code?
No.
>
> *** What else should I do?
>
Send the WordNetProcessor code and, as rand note, stop using
scriptlets in jsps :-)
> BTW, Since I use Virtuozzo powered VPS for its hosting, it seems that
> Virtuozzo lower the number of available file descriptors since most
> likely they are shared among all virtual private servers.
>
> Thanks in advance for any advice / comment.
>
>
> --
> Mladen Adamovic
regards
Leon
> http://www.online-utility.org
> http://www.cheapvps.info
> http://www.vpsreview.com
>
>
>
>
>
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org