You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Christopher Schultz <ch...@christopherschultz.net> on 2009/05/21 17:55:43 UTC
NIO Connector: Too many open files
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
All,
I've been testing the performance of various Tomcat configurations
against Apache httpd and my serious tests are not completing for the NIO
connector because the server is running out of files:
> May 20, 2009 2:35:55 AM org.apache.tomcat.util.net.NioEndpoint$Acceptor run
> SEVERE: Socket accept failed
> java.io.IOException: Too many open files
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
> at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
> at java.lang.Thread.run(Thread.java:619)
A bit of background for those who haven't followed the "Apache httpd vs
Tomcat static content performance" thread:
I'm running Tomcat 6.0.18 using tcnative 1.1.16. Apache httpd is not
being used for this test, so the client is contacting Tomcat directly
from localhost.
$ uname -a
Linux chadis 2.6.14-gentoo-r5 #2 PREEMPT Sat Dec 17 16:30:55 EST 2005
i686 AMD Athlon(tm) XP 1700+ AuthenticAMD GNU/Linux
$ java -version
java version "1.6.0_13"
Java(TM) SE Runtime Environment (build 1.6.0_13-b03)
Java HotSpot(TM) Client VM (build 11.3-b02, mixed mode, sharing)
$ ulimit -n (fds per process limit)
1024
1GiB RAM on the machine, here are the heap details /after/ the tests
are run:
$ jmap -heap 1430
Attaching to process ID 1430, please wait...
Debugger attached successfully.
Client compiler detected.
JVM version is 11.3-b02
using thread-local object allocation.
Mark Sweep Compact GC
Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 67108864 (64.0MB)
NewSize = 1048576 (1.0MB)
MaxNewSize = 4294901760 (4095.9375MB)
OldSize = 4194304 (4.0MB)
NewRatio = 12
SurvivorRatio = 8
PermSize = 12582912 (12.0MB)
MaxPermSize = 67108864 (64.0MB)
Heap Usage:
New Generation (Eden + 1 Survivor Space):
capacity = 2228224 (2.125MB)
used = 612888 (0.5844955444335938MB)
free = 1615336 (1.5405044555664062MB)
27.505672679227942% used
Eden Space:
capacity = 2031616 (1.9375MB)
used = 612888 (0.5844955444335938MB)
free = 1418728 (1.3530044555664062MB)
30.167511970766128% used
- From Space:
capacity = 196608 (0.1875MB)
used = 0 (0.0MB)
free = 196608 (0.1875MB)
0.0% used
To Space:
capacity = 196608 (0.1875MB)
used = 0 (0.0MB)
free = 196608 (0.1875MB)
0.0% used
tenured generation:
capacity = 28311552 (27.0MB)
used = 20464784 (19.516738891601562MB)
free = 7846768 (7.4832611083984375MB)
72.28421811704283% used
Perm Generation:
capacity = 12582912 (12.0MB)
used = 8834304 (8.425048828125MB)
free = 3748608 (3.574951171875MB)
70.208740234375% used
Here are my <Connector> configurations:
<!-- Regular non-APR Coyote Connector -->
<Connector port="8001"
protocol="org.apache.coyote.http11.Http11Protocol"
connectionTimeout="20000"
server="Coyote1.1non-APR"
/>
<!-- APR Connector -->
<Connector port="8002"
protocol="org.apache.coyote.http11.Http11AprProtocol"
useSendfile="true"
connectionTimeout="20000"
server="Coyote1.1APR"
/>
<!-- APR without sendfile -->
<Connector port="8003"
protocol="org.apache.coyote.http11.Http11AprProtocol"
useSendfile="false"
connectionTimeout="20000"
server="Coyote1.1APRw/osendfile"
/>
<!-- NIO Connector -->
<Connector port="8004"
protocol="org.apache.coyote.http11.Http11NioProtocol"
useSendfile="true"
connectionTimeout="20000"
server="Coyote1.1NIO"
/>
<!-- APR without sendfile -->
<Connector port="8005"
protocol="org.apache.coyote.http11.Http11NioProtocol"
useSendfile="false"
connectionTimeout="20000"
server="Coyote1.1NIOw/osendfile"
/>
All connectors are configured at once, so I should have a maximum of 40
threads in each pool. The command I ran to benchmark each connector was
(for example):
/usr/sbin/ab -c 40 -t 480 -n 10000000 http://localhost:8004/4kiB.bin
This runs ApacheBench for 8 minutes with 40 client threads requesting a
4k file over and over again. This particular test succeeded, but there
are 14 more tests, each using a file twice the size of the previous
test. After the 128k file test, every single test fails after that.
The last test I ran (with only 1 thread instead of 40), the NIO
connector died in the same way, but the NIO connector without sendfile
enabled appeared to work properly. This time (40 threads), neither of
the connectors worker properly, the NIO connector failing to complete
any tests after the 128kb test and the NIO-sendfile connector failed to
complete /all/ of the tests (automatically run immediately following the
NIO tests).
No OOMEs were encountered: only the exception shown above (no more
files). On my previous tests, lsof reported that only one of my files
was still open by the process. After this most recent test, it appears
that 954 of my static files are still open by the process (and the test
ended over 24 hours ago).
The initial set of tests (c=1) seemed to recover, while this second set
of tests (c=40) has not.
My knee-jerk reaction is that the most number of files that should ever
be open is 40: one per request processing thread. Something, somewhere
is causing these file descriptors to stay open.
Unfortunately, I don't have any GC information for the time period
covering the test.
I still have the JVM running, so I can probably inspect it for certain
things if anyone has any questions. Unfortunately, I can't run any new
JSP files (out of files!) and it looks like I can't connect using
jconsole (probably because the JVM can't open a new socket).
I'd love some suggestions at to what's going on, here.
Thanks,
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkoVeX8ACgkQ9CaO5/Lv0PCN3QCfXV3tu/BIM7I8/dCvQ0ehkVws
s/QAn3Kf4PjoAdSnUtp7FGMLz5u1W0FY
=lz7Y
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: NIO Connector: Too many open files
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Rainer,
On 5/22/2009 8:48 AM, Rainer Jung wrote:
> The file descriptos thing is totaly independent. I hijacked the thread :)
Yeah, I know. Filip independently mentioned that there's a fd leak in
the NIO implementation includes in 6.0.18. I have yet to test the fix,
but I'm hopeful.
Thanks,
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkoW+fsACgkQ9CaO5/Lv0PBGBQCgnlcDb+h+LZ1/qggGuyLq+P77
gvwAoKy3tPXilE+4poxrhl9wRLE7XQaK
=PmtD
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: NIO Connector: Too many open files
Posted by Rainer Jung <ra...@kippdata.de>.
Hi Chris,
On 22.05.2009 14:14, Christopher Schultz wrote:
> Rainer,
>
> On 5/21/2009 12:21 PM, Rainer Jung wrote:
>> 2 remarks about all your stress testing efforts:
>
>> A) TIME_WAIT
>
>> When not doing HTTP Keep-Alive, under high load the size of the TCP hash
>> table and the effectiveness of the system to lookp up TCP connections
>> can limit the throughput you can reach. More precisely, depending on the
>> excat way of connection shutdown, you get TIME_WAIT states for the
>> finished connections (without HTTP Keep Alive it could be one such
>> connection per request). Most systems get slow, once the number of those
>> connections reaches somthing arounf 30000.
>
> That's fine, but the TIME_WAIT connections should be counted against the
> process's file limit, should it? At that point, the process has released
> the connection and the OS is babysitting it through the final stages of
> TCP shutdown.
Those connections will *not* be counted against process file
descriptors. They only exist as an entry in a TCP connection table. They
are no longer associated with the process. It's more of a TCP house
cleaning thing.
> I understand that, with keepalive disabled, performance will kind of
> suck. But, I shouldn't be running out of file descriptors.
Not our of FDs, but if the number of TIME_WAITs gets huge (check via
netstat during the run), your TCP throughput will drop and will be
restricted by the size of the connection hash.
>> E.g. if you are doing 2000 requests per second without HTTP Keep Alive
>> and the combination of web server and stress test tool leads to
>> TIME_WAITs, after 15 seconds your table size might reach a critical size.
>
> Meaning that the kernel can't keep up, or the NIO connector can't keep
> up? I suspect the latter, because the other tests under the same
> conditions at least complete... the NIO one appears not to have a
> chance. Now, I'm running 6 tests and the NIO test is the 5th one, so
> it's possible that it's just poorly positioned in my test batter. But,
> since I've observed this failure at essentially the same place each
> time, I suspect the NIO connector itself is at fault.
I'm talking about a very general TCP thing. I'm not saying you actually
ran into it, but I'm saying that it makes sense to check the number of
TIME_WAITs via netstat during the test. If it gets very big, than the
TCP implementation will limit your throughput and most likely will
become the first bottleneck you hit. Again: I'm not saying that already
happened, but you should check, whether you run into this while doing
the test.
>> Not using HTTP Keep Alive will very likely limit quickly the achievable
>> throughput when going up in concurrency.
>
> I'm willing to accept that, but 40 max connections should not be
> resulting in hundreds of left-open file descriptors.
The file descriptos thing is totaly independent. I hijacked the thread :)
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: NIO Connector: Too many open files
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Rainer,
On 5/21/2009 12:21 PM, Rainer Jung wrote:
> 2 remarks about all your stress testing efforts:
>
> A) TIME_WAIT
>
> When not doing HTTP Keep-Alive, under high load the size of the TCP hash
> table and the effectiveness of the system to lookp up TCP connections
> can limit the throughput you can reach. More precisely, depending on the
> excat way of connection shutdown, you get TIME_WAIT states for the
> finished connections (without HTTP Keep Alive it could be one such
> connection per request). Most systems get slow, once the number of those
> connections reaches somthing arounf 30000.
That's fine, but the TIME_WAIT connections should be counted against the
process's file limit, should it? At that point, the process has released
the connection and the OS is babysitting it through the final stages of
TCP shutdown.
I understand that, with keepalive disabled, performance will kind of
suck. But, I shouldn't be running out of file descriptors.
> E.g. if you are doing 2000 requests per second without HTTP Keep Alive
> and the combination of web server and stress test tool leads to
> TIME_WAITs, after 15 seconds your table size might reach a critical size.
Meaning that the kernel can't keep up, or the NIO connector can't keep
up? I suspect the latter, because the other tests under the same
conditions at least complete... the NIO one appears not to have a
chance. Now, I'm running 6 tests and the NIO test is the 5th one, so
it's possible that it's just poorly positioned in my test batter. But,
since I've observed this failure at essentially the same place each
time, I suspect the NIO connector itself is at fault.
> Not using HTTP Keep Alive will very likely limit quickly the achievable
> throughput when going up in concurrency.
I'm willing to accept that, but 40 max connections should not be
resulting in hundreds of left-open file descriptors.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkoWlzUACgkQ9CaO5/Lv0PDtKgCfc9ZOaqcHfvdJHD28wZIiwMAq
52EAn3N1Nk3JoqSOcDkjdlsmJ49t+Yfr
=FQdw
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: NIO Connector: Too many open files
Posted by Rainer Jung <ra...@kippdata.de>.
2 remarks about all your stress testing efforts:
A) TIME_WAIT
When not doing HTTP Keep-Alive, under high load the size of the TCP hash
table and the effectiveness of the system to lookp up TCP connections
can limit the throughput you can reach. More precisely, depending on the
excat way of connection shutdown, you get TIME_WAIT states for the
finished connections (without HTTP Keep Alive it could be one such
connection per request). Most systems get slow, once the number of those
connections reaches somthing arounf 30000.
E.g. if you are doing 2000 requests per second without HTTP Keep Alive
and the combination of web server and stress test tool leads to
TIME_WAITs, after 15 seconds your table size might reach a critical size.
The amount of time a system waits before it destroys TIME_WAIT
connections varies. Solaris 4 minutes, but tunable down to 5 seconds,
Linux 1 minute (I think) and still not tunable (a free pizza to the
first one who can show me how to tune the time interval it takes for a
TCP connection to be moved out of TIME_WAIT), on Windows I think also 1
minute but tunable.
Not using HTTP Keep Alive will very likely limit quickly the achievable
throughput when going up in concurrency.
B) Resource usage
You are doing a first interesting analysis, namely the base throughput
you can reach using concurrency one. Now the throughput we can reach
using a given setup always depends on the first bottleneck we hit.
For big files it might be the network bandwidth. For small files it
might be CPU or maybe I/Os per second we can do on the interface.
So it is also interesting to compare the resource usage. Some resources
are harder to measure (like memory), but the one easy to measure
resource is CPU. So you get some more info, when also measuring CPU per
request. Think of it as maximum speed and gas efficiency.
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: NIO Connector: Too many open files
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Rainer,
On 5/22/2009 10:32 AM, Rainer Jung wrote:
> On 22.05.2009 15:46, Christopher Schultz wrote:
>> Rainer,
>>
>> On 5/22/2009 8:55 AM, Rainer Jung wrote:
>>> You could run a JSP including a call to System.gc();
>> Right. The JVM is out of file descriptors. I cannot deploy a new JSP
>> onto the server without restarting it. ;) I mentioned this in my initial
>> post.
>
> Yes, of course. In that case you are pretty much restricted to the
> outside tools you already used. So it would make sense to deploy and
> precompile/call the new JSP once before starting the next run, so you at
> least have a chance of calling it.
Yeah, I had one of those deployed already. Unfortunately, one of the
things that JSP does is call InetAddress.getLocalHost() which fails
because ... I was out of file descriptors. It's amazing how much stuff
stops working once you hit that wall and nothing will be released ;)
> Of course the connection the hte http
> connector needed for the request also needs an FD, but I had the
> impression from your last post, that a few of those FDs are being closed
> after some time. Maybe enough for calling the JSP.
Yeah, I could make an HTTP connection to the server, but I think that's
because the connector already had an fd allocated for the listen.
Actually, I'm not sure why some things worked (like making new requests)
but others did not (such as doing a localhost lookup).
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkoW+bIACgkQ9CaO5/Lv0PB2UQCeNTDx5CBnaU3F44sdchJZ29gZ
OUEAoJVlwqaRoig1HLi4fBra2QL5JyhU
=9czx
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: NIO Connector: Too many open files
Posted by Rainer Jung <ra...@kippdata.de>.
On 22.05.2009 15:46, Christopher Schultz wrote:
> Rainer,
>
> On 5/22/2009 8:55 AM, Rainer Jung wrote:
>> You could run a JSP including a call to System.gc();
>
> Right. The JVM is out of file descriptors. I cannot deploy a new JSP
> onto the server without restarting it. ;) I mentioned this in my initial
> post.
Yes, of course. In that case you are pretty much restricted to the
outside tools you already used. So it would make sense to deploy and
precompile/call the new JSP once before starting the next run, so you at
least have a chance of calling it. Of course the connection the hte http
connector needed for the request also needs an FD, but I had the
impression from your last post, that a few of those FDs are being closed
after some time. Maybe enough for calling the JSP.
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: NIO Connector: Too many open files
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Rainer,
On 5/22/2009 8:55 AM, Rainer Jung wrote:
> You could run a JSP including a call to System.gc();
Right. The JVM is out of file descriptors. I cannot deploy a new JSP
onto the server without restarting it. ;) I mentioned this in my initial
post.
> First of all this really looks bad and interesting. So we would benefit
> from understanding what's going on.
Filip indicates that there is a resource leak fixed in the NIO
connector. I'll have to build that and upgrade my TC installation to see
if it fixes this problem.
> Are there chances you can try again and see, whether those additional
> FDs for the static content go way if ou trigger a major GC, either via
> such a JSP, or via the JConsole or any other way you like?
I've tried everything I can think of. jconsole couldn't connect (still
out of file descriptors!) which I also already mentioned. I gave up a
few minutes ago and stopped the server in preparation for upgrading the
connectors themselves.
I'll reply once I have some more information.
Thanks,
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: NIO Connector: Too many open files
Posted by Rainer Jung <ra...@kippdata.de>.
Hi Chris,
On 22.05.2009 14:29, Christopher Schultz wrote:
>>> $ jmap -heap 1430
>>> Attaching to process ID 1430, please wait...
>>> Debugger attached successfully.
>>> Client compiler detected.
>>> JVM version is 11.3-b02
>>>
>>> using thread-local object allocation.
>>> Mark Sweep Compact GC
>>>
>>> Heap Configuration:
>>> MinHeapFreeRatio = 40
>>> MaxHeapFreeRatio = 70
>>> MaxHeapSize = 67108864 (64.0MB)
>>> NewSize = 1048576 (1.0MB)
>>> MaxNewSize = 4294901760 (4095.9375MB)
>>> OldSize = 4194304 (4.0MB)
>>> NewRatio = 12
>>> SurvivorRatio = 8
>>> PermSize = 12582912 (12.0MB)
>>> MaxPermSize = 67108864 (64.0MB)
>>>
>>> Heap Usage:
>>> New Generation (Eden + 1 Survivor Space):
>>> capacity = 2228224 (2.125MB)
>>> used = 612888 (0.5844955444335938MB)
>>> free = 1615336 (1.5405044555664062MB)
>>> 27.505672679227942% used
>>> Eden Space:
>>> capacity = 2031616 (1.9375MB)
>>> used = 612888 (0.5844955444335938MB)
>>> free = 1418728 (1.3530044555664062MB)
>>> 30.167511970766128% used
>>> - From Space:
>>> capacity = 196608 (0.1875MB)
>>> used = 0 (0.0MB)
>>> free = 196608 (0.1875MB)
>>> 0.0% used
>>> To Space:
>>> capacity = 196608 (0.1875MB)
>>> used = 0 (0.0MB)
>>> free = 196608 (0.1875MB)
>>> 0.0% used
>>> tenured generation:
>>> capacity = 28311552 (27.0MB)
>>> used = 20464784 (19.516738891601562MB)
>>> free = 7846768 (7.4832611083984375MB)
>>> 72.28421811704283% used
>>> Perm Generation:
>>> capacity = 12582912 (12.0MB)
>>> used = 8834304 (8.425048828125MB)
>>> free = 3748608 (3.574951171875MB)
>>> 70.208740234375% used
>>>
>>> Here are my <Connector> configurations:
>>>
>>> <!-- Regular non-APR Coyote Connector -->
>>> <Connector port="8001"
>>> protocol="org.apache.coyote.http11.Http11Protocol"
>>> connectionTimeout="20000"
>>> server="Coyote1.1non-APR"
>>> />
>>>
>>> <!-- APR Connector -->
>>> <Connector port="8002"
>>> protocol="org.apache.coyote.http11.Http11AprProtocol"
>>> useSendfile="true"
>>> connectionTimeout="20000"
>>> server="Coyote1.1APR"
>>> />
>>>
>>> <!-- APR without sendfile -->
>>> <Connector port="8003"
>>> protocol="org.apache.coyote.http11.Http11AprProtocol"
>>> useSendfile="false"
>>> connectionTimeout="20000"
>>> server="Coyote1.1APRw/osendfile"
>>> />
>>>
>>> <!-- NIO Connector -->
>>> <Connector port="8004"
>>> protocol="org.apache.coyote.http11.Http11NioProtocol"
>>> useSendfile="true"
>>> connectionTimeout="20000"
>>> server="Coyote1.1NIO"
>>> />
>>>
>>> <!-- APR without sendfile -->
>>> <Connector port="8005"
>>> protocol="org.apache.coyote.http11.Http11NioProtocol"
>>> useSendfile="false"
>>> connectionTimeout="20000"
>>> server="Coyote1.1NIOw/osendfile"
>>> />
>>>
>>> All connectors are configured at once, so I should have a maximum of 40
>>> threads in each pool. The command I ran to benchmark each connector was
>>> (for example):
>>>
>>> /usr/sbin/ab -c 40 -t 480 -n 10000000 http://localhost:8004/4kiB.bin
>>>
>>> This runs ApacheBench for 8 minutes with 40 client threads requesting a
>>> 4k file over and over again. This particular test succeeded, but there
>>> are 14 more tests, each using a file twice the size of the previous
>>> test. After the 128k file test, every single test fails after that.
>>>
>>> The last test I ran (with only 1 thread instead of 40), the NIO
>>> connector died in the same way, but the NIO connector without sendfile
>>> enabled appeared to work properly. This time (40 threads), neither of
>>> the connectors worker properly, the NIO connector failing to complete
>>> any tests after the 128kb test and the NIO-sendfile connector failed to
>>> complete /all/ of the tests (automatically run immediately following the
>>> NIO tests).
>>>
>>> No OOMEs were encountered: only the exception shown above (no more
>>> files). On my previous tests, lsof reported that only one of my files
>>> was still open by the process. After this most recent test, it appears
>>> that 954 of my static files are still open by the process (and the test
>>> ended over 24 hours ago).
>>>
>>> The initial set of tests (c=1) seemed to recover, while this second set
>>> of tests (c=40) has not.
>>>
>>> My knee-jerk reaction is that the most number of files that should ever
>>> be open is 40: one per request processing thread. Something, somewhere
>>> is causing these file descriptors to stay open.
>>>
>>> Unfortunately, I don't have any GC information for the time period
>>> covering the test.
>>>
>>> I still have the JVM running, so I can probably inspect it for certain
>>> things if anyone has any questions. Unfortunately, I can't run any new
>>> JSP files (out of files!) and it looks like I can't connect using
>>> jconsole (probably because the JVM can't open a new socket).
>>>
>>> I'd love some suggestions at to what's going on, here.
>> Maybe looking at the directory /proc/PID/fd (sorry PID) will give more
>> info (at least of the FDs are still in use). Or you might have "lsof"
>> inspected. Sometimes proc is enough, sometimes you need lsof to find out
>> more.
>
> I have already have lsof information for the process:
>
> $ lsof -p 1430 | grep 'ROOT/[0-9A-Za-z]*\.bin' | wc
> 954 8586 116376
>
> Note that this is now 48 hours or so after the test completed, so I
> don't think I'm just waiting around for a GC to occur (then again,
> there's probably little in the way of activity going on, so maybe I /am/
> just waiting around for a GC). I am able to execute a JSP on the server
> side that gives me memory information... I ran it like 100 times to
> watch the available memory decrease, then recover -- probably from a
> minor GC. Maybe a full GC is required. Maybe the interaction between the
> NIO connector and tcnative results in leaked file descriptors.
You could run a JSP including a call to System.gc();
If you don't have DisableExplicitGC set, then usually this triggers a
major GC (it's not 100% sure, but until now all JVMs I saw start a major
GC). You could call the JSP two times, and then check via your aother
JSP, whether the memory numbers actually have changed.
> This is what the raw output looks like (abbreviated, of course):
>
> COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
> java 1430 chris 68r REG 3,3 262144 8751655
> /home/chris/app/connector-test/8785/webapps/ROOT/256kiB.bin
> java 1430 chris 70r REG 3,3 262144 8751655
> /home/chris/app/connector-test/8785/webapps/ROOT/256kiB.bin
> java 1430 chris 71r REG 3,3 262144 8751655
> /home/chris/app/connector-test/8785/webapps/ROOT/256kiB.bin
> java 1430 chris 72r REG 3,3 262144 8751655
> /home/chris/app/connector-test/8785/webapps/ROOT/256kiB.bin
> java 1430 chris 73r REG 3,3 262144 8751655
> /home/chris/app/connector-test/8785/webapps/ROOT/256kiB.bin
> java 1430 chris 74r REG 3,3 262144 8751655
> /home/chris/app/connector-test/8785/webapps/ROOT/256kiB.bin
> java 1430 chris 75r REG 3,3 262144 8751655
> /home/chris/app/connector-test/8785/webapps/ROOT/256kiB.bin
> java 1430 chris 76r REG 3,3 262144 8751655
> /home/chris/app/connector-test/8785/webapps/ROOT/256kiB.bin
> java 1430 chris 77r REG 3,3 262144 8751655
> /home/chris/app/connector-test/8785/webapps/ROOT/256kiB.bin
> java 1430 chris 78r REG 3,3 262144 8751655
> /home/chris/app/connector-test/8785/webapps/ROOT/256kiB.bin
> java 1430 chris 79r REG 3,3 262144 8751655
> /home/chris/app/connector-test/8785/webapps/ROOT/256kiB.bin
> java 1430 chris 80r REG 3,3 262144 8751655
> /home/chris/app/connector-test/8785/webapps/ROOT/256kiB.bin
> java 1430 chris 81r REG 3,3 262144 8751655
> /home/chris/app/connector-test/8785/webapps/ROOT/256kiB.bin
> java 1430 chris 82r REG 3,3 262144 8751655
> /home/chris/app/connector-test/8785/webapps/ROOT/256kiB.bin
> java 1430 chris 83r REG 3,3 262144 8751655
> /home/chris/app/connector-test/8785/webapps/ROOT/256kiB.bin
>
> etc.
>
> It's not just the 256KiB files that are still open (those were the first
> file size test to fail to complete, then all the rest failed to complete
> as well), though that file represents most of them:
> $ lsof -p 1430 | grep 'ROOT/[0-9A-Za-z]*\.bin' | sed -e 's/[^/]\+//' |
> sort | uniq -c
> 6 /home/chris/app/connector-test/8785/webapps/ROOT/1MiB.bin
> 948 /home/chris/app/connector-test/8785/webapps/ROOT/256kiB.bin
>
> Is there any way to forceably close a file handle on Linux from outside
> the process? I wasn't able to find a way to do this, or I would have
> tried to force a full GC.
First of all this really looks bad and interesting. So we would benefit
from understanding what's going on.
Either we have a bug somewhere, or it is really some finalizer that
needs to run in order to close the file. I would expect the finalizer to
be run only when the object is actually garbage collected, so if the
object moved to the tenured space, a minor GC will not be enough to get
it collected. If there is no load Tomcat only allocated very few new
objects (like when checking for changes of web.xml etc., things Tomcat
does even when no requests come in) and it is unlikely that a major GC
(GC of tenured) happens without load. So that's why the above JSP with
System.gc() could help.
Are there chances you can try again and see, whether those additional
FDs for the static content go way if ou trigger a major GC, either via
such a JSP, or via the JConsole or any other way you like?
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: NIO Connector: Too many open files
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Rainer,
On 5/21/2009 12:13 PM, Rainer Jung wrote:
> On 21.05.2009 17:55, Christopher Schultz wrote:
>> All,
>>
>> I've been testing the performance of various Tomcat configurations
>> against Apache httpd and my serious tests are not completing for the NIO
>> connector because the server is running out of files:
>>
>>> May 20, 2009 2:35:55 AM org.apache.tomcat.util.net.NioEndpoint$Acceptor run
>>> SEVERE: Socket accept failed
>>> java.io.IOException: Too many open files
>>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
>>> at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
>>> at java.lang.Thread.run(Thread.java:619)
>> A bit of background for those who haven't followed the "Apache httpd vs
>> Tomcat static content performance" thread:
>>
>> I'm running Tomcat 6.0.18 using tcnative 1.1.16. Apache httpd is not
>> being used for this test, so the client is contacting Tomcat directly
>> from localhost.
>>
>> $ uname -a
>> Linux chadis 2.6.14-gentoo-r5 #2 PREEMPT Sat Dec 17 16:30:55 EST 2005
>> i686 AMD Athlon(tm) XP 1700+ AuthenticAMD GNU/Linux
>>
>> $ java -version
>> java version "1.6.0_13"
>> Java(TM) SE Runtime Environment (build 1.6.0_13-b03)
>> Java HotSpot(TM) Client VM (build 11.3-b02, mixed mode, sharing)
>>
>> $ ulimit -n (fds per process limit)
>> 1024
>>
>> 1GiB RAM on the machine, here are the heap details /after/ the tests
>> are run:
>>
>> $ jmap -heap 1430
>> Attaching to process ID 1430, please wait...
>> Debugger attached successfully.
>> Client compiler detected.
>> JVM version is 11.3-b02
>>
>> using thread-local object allocation.
>> Mark Sweep Compact GC
>>
>> Heap Configuration:
>> MinHeapFreeRatio = 40
>> MaxHeapFreeRatio = 70
>> MaxHeapSize = 67108864 (64.0MB)
>> NewSize = 1048576 (1.0MB)
>> MaxNewSize = 4294901760 (4095.9375MB)
>> OldSize = 4194304 (4.0MB)
>> NewRatio = 12
>> SurvivorRatio = 8
>> PermSize = 12582912 (12.0MB)
>> MaxPermSize = 67108864 (64.0MB)
>>
>> Heap Usage:
>> New Generation (Eden + 1 Survivor Space):
>> capacity = 2228224 (2.125MB)
>> used = 612888 (0.5844955444335938MB)
>> free = 1615336 (1.5405044555664062MB)
>> 27.505672679227942% used
>> Eden Space:
>> capacity = 2031616 (1.9375MB)
>> used = 612888 (0.5844955444335938MB)
>> free = 1418728 (1.3530044555664062MB)
>> 30.167511970766128% used
>> - From Space:
>> capacity = 196608 (0.1875MB)
>> used = 0 (0.0MB)
>> free = 196608 (0.1875MB)
>> 0.0% used
>> To Space:
>> capacity = 196608 (0.1875MB)
>> used = 0 (0.0MB)
>> free = 196608 (0.1875MB)
>> 0.0% used
>> tenured generation:
>> capacity = 28311552 (27.0MB)
>> used = 20464784 (19.516738891601562MB)
>> free = 7846768 (7.4832611083984375MB)
>> 72.28421811704283% used
>> Perm Generation:
>> capacity = 12582912 (12.0MB)
>> used = 8834304 (8.425048828125MB)
>> free = 3748608 (3.574951171875MB)
>> 70.208740234375% used
>>
>> Here are my <Connector> configurations:
>>
>> <!-- Regular non-APR Coyote Connector -->
>> <Connector port="8001"
>> protocol="org.apache.coyote.http11.Http11Protocol"
>> connectionTimeout="20000"
>> server="Coyote1.1non-APR"
>> />
>>
>> <!-- APR Connector -->
>> <Connector port="8002"
>> protocol="org.apache.coyote.http11.Http11AprProtocol"
>> useSendfile="true"
>> connectionTimeout="20000"
>> server="Coyote1.1APR"
>> />
>>
>> <!-- APR without sendfile -->
>> <Connector port="8003"
>> protocol="org.apache.coyote.http11.Http11AprProtocol"
>> useSendfile="false"
>> connectionTimeout="20000"
>> server="Coyote1.1APRw/osendfile"
>> />
>>
>> <!-- NIO Connector -->
>> <Connector port="8004"
>> protocol="org.apache.coyote.http11.Http11NioProtocol"
>> useSendfile="true"
>> connectionTimeout="20000"
>> server="Coyote1.1NIO"
>> />
>>
>> <!-- APR without sendfile -->
>> <Connector port="8005"
>> protocol="org.apache.coyote.http11.Http11NioProtocol"
>> useSendfile="false"
>> connectionTimeout="20000"
>> server="Coyote1.1NIOw/osendfile"
>> />
>>
>> All connectors are configured at once, so I should have a maximum of 40
>> threads in each pool. The command I ran to benchmark each connector was
>> (for example):
>>
>> /usr/sbin/ab -c 40 -t 480 -n 10000000 http://localhost:8004/4kiB.bin
>>
>> This runs ApacheBench for 8 minutes with 40 client threads requesting a
>> 4k file over and over again. This particular test succeeded, but there
>> are 14 more tests, each using a file twice the size of the previous
>> test. After the 128k file test, every single test fails after that.
>>
>> The last test I ran (with only 1 thread instead of 40), the NIO
>> connector died in the same way, but the NIO connector without sendfile
>> enabled appeared to work properly. This time (40 threads), neither of
>> the connectors worker properly, the NIO connector failing to complete
>> any tests after the 128kb test and the NIO-sendfile connector failed to
>> complete /all/ of the tests (automatically run immediately following the
>> NIO tests).
>>
>> No OOMEs were encountered: only the exception shown above (no more
>> files). On my previous tests, lsof reported that only one of my files
>> was still open by the process. After this most recent test, it appears
>> that 954 of my static files are still open by the process (and the test
>> ended over 24 hours ago).
>>
>> The initial set of tests (c=1) seemed to recover, while this second set
>> of tests (c=40) has not.
>>
>> My knee-jerk reaction is that the most number of files that should ever
>> be open is 40: one per request processing thread. Something, somewhere
>> is causing these file descriptors to stay open.
>>
>> Unfortunately, I don't have any GC information for the time period
>> covering the test.
>>
>> I still have the JVM running, so I can probably inspect it for certain
>> things if anyone has any questions. Unfortunately, I can't run any new
>> JSP files (out of files!) and it looks like I can't connect using
>> jconsole (probably because the JVM can't open a new socket).
>>
>> I'd love some suggestions at to what's going on, here.
>
> Maybe looking at the directory /proc/PID/fd (sorry PID) will give more
> info (at least of the FDs are still in use). Or you might have "lsof"
> inspected. Sometimes proc is enough, sometimes you need lsof to find out
> more.
I have already have lsof information for the process:
$ lsof -p 1430 | grep 'ROOT/[0-9A-Za-z]*\.bin' | wc
954 8586 116376
Note that this is now 48 hours or so after the test completed, so I
don't think I'm just waiting around for a GC to occur (then again,
there's probably little in the way of activity going on, so maybe I /am/
just waiting around for a GC). I am able to execute a JSP on the server
side that gives me memory information... I ran it like 100 times to
watch the available memory decrease, then recover -- probably from a
minor GC. Maybe a full GC is required. Maybe the interaction between the
NIO connector and tcnative results in leaked file descriptors.
This is what the raw output looks like (abbreviated, of course):
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
java 1430 chris 68r REG 3,3 262144 8751655
/home/chris/app/connector-test/8785/webapps/ROOT/256kiB.bin
java 1430 chris 70r REG 3,3 262144 8751655
/home/chris/app/connector-test/8785/webapps/ROOT/256kiB.bin
java 1430 chris 71r REG 3,3 262144 8751655
/home/chris/app/connector-test/8785/webapps/ROOT/256kiB.bin
java 1430 chris 72r REG 3,3 262144 8751655
/home/chris/app/connector-test/8785/webapps/ROOT/256kiB.bin
java 1430 chris 73r REG 3,3 262144 8751655
/home/chris/app/connector-test/8785/webapps/ROOT/256kiB.bin
java 1430 chris 74r REG 3,3 262144 8751655
/home/chris/app/connector-test/8785/webapps/ROOT/256kiB.bin
java 1430 chris 75r REG 3,3 262144 8751655
/home/chris/app/connector-test/8785/webapps/ROOT/256kiB.bin
java 1430 chris 76r REG 3,3 262144 8751655
/home/chris/app/connector-test/8785/webapps/ROOT/256kiB.bin
java 1430 chris 77r REG 3,3 262144 8751655
/home/chris/app/connector-test/8785/webapps/ROOT/256kiB.bin
java 1430 chris 78r REG 3,3 262144 8751655
/home/chris/app/connector-test/8785/webapps/ROOT/256kiB.bin
java 1430 chris 79r REG 3,3 262144 8751655
/home/chris/app/connector-test/8785/webapps/ROOT/256kiB.bin
java 1430 chris 80r REG 3,3 262144 8751655
/home/chris/app/connector-test/8785/webapps/ROOT/256kiB.bin
java 1430 chris 81r REG 3,3 262144 8751655
/home/chris/app/connector-test/8785/webapps/ROOT/256kiB.bin
java 1430 chris 82r REG 3,3 262144 8751655
/home/chris/app/connector-test/8785/webapps/ROOT/256kiB.bin
java 1430 chris 83r REG 3,3 262144 8751655
/home/chris/app/connector-test/8785/webapps/ROOT/256kiB.bin
etc.
It's not just the 256KiB files that are still open (those were the first
file size test to fail to complete, then all the rest failed to complete
as well), though that file represents most of them:
$ lsof -p 1430 | grep 'ROOT/[0-9A-Za-z]*\.bin' | sed -e 's/[^/]\+//' |
sort | uniq -c
6 /home/chris/app/connector-test/8785/webapps/ROOT/1MiB.bin
948 /home/chris/app/connector-test/8785/webapps/ROOT/256kiB.bin
Is there any way to forceably close a file handle on Linux from outside
the process? I wasn't able to find a way to do this, or I would have
tried to force a full GC.
Thanks,
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkoWmqsACgkQ9CaO5/Lv0PCI9QCfa84sSCod/2bQ0hewsbSA+BD5
vX0AnRbF4cL/11B3Fl9Rrj13OawhmP//
=/eOP
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: NIO Connector: Too many open files
Posted by Rainer Jung <ra...@kippdata.de>.
On 21.05.2009 17:55, Christopher Schultz wrote:
> All,
>
> I've been testing the performance of various Tomcat configurations
> against Apache httpd and my serious tests are not completing for the NIO
> connector because the server is running out of files:
>
>> May 20, 2009 2:35:55 AM org.apache.tomcat.util.net.NioEndpoint$Acceptor run
>> SEVERE: Socket accept failed
>> java.io.IOException: Too many open files
>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
>> at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
>> at java.lang.Thread.run(Thread.java:619)
>
> A bit of background for those who haven't followed the "Apache httpd vs
> Tomcat static content performance" thread:
>
> I'm running Tomcat 6.0.18 using tcnative 1.1.16. Apache httpd is not
> being used for this test, so the client is contacting Tomcat directly
> from localhost.
>
> $ uname -a
> Linux chadis 2.6.14-gentoo-r5 #2 PREEMPT Sat Dec 17 16:30:55 EST 2005
> i686 AMD Athlon(tm) XP 1700+ AuthenticAMD GNU/Linux
>
> $ java -version
> java version "1.6.0_13"
> Java(TM) SE Runtime Environment (build 1.6.0_13-b03)
> Java HotSpot(TM) Client VM (build 11.3-b02, mixed mode, sharing)
>
> $ ulimit -n (fds per process limit)
> 1024
>
> 1GiB RAM on the machine, here are the heap details /after/ the tests
> are run:
>
> $ jmap -heap 1430
> Attaching to process ID 1430, please wait...
> Debugger attached successfully.
> Client compiler detected.
> JVM version is 11.3-b02
>
> using thread-local object allocation.
> Mark Sweep Compact GC
>
> Heap Configuration:
> MinHeapFreeRatio = 40
> MaxHeapFreeRatio = 70
> MaxHeapSize = 67108864 (64.0MB)
> NewSize = 1048576 (1.0MB)
> MaxNewSize = 4294901760 (4095.9375MB)
> OldSize = 4194304 (4.0MB)
> NewRatio = 12
> SurvivorRatio = 8
> PermSize = 12582912 (12.0MB)
> MaxPermSize = 67108864 (64.0MB)
>
> Heap Usage:
> New Generation (Eden + 1 Survivor Space):
> capacity = 2228224 (2.125MB)
> used = 612888 (0.5844955444335938MB)
> free = 1615336 (1.5405044555664062MB)
> 27.505672679227942% used
> Eden Space:
> capacity = 2031616 (1.9375MB)
> used = 612888 (0.5844955444335938MB)
> free = 1418728 (1.3530044555664062MB)
> 30.167511970766128% used
> - From Space:
> capacity = 196608 (0.1875MB)
> used = 0 (0.0MB)
> free = 196608 (0.1875MB)
> 0.0% used
> To Space:
> capacity = 196608 (0.1875MB)
> used = 0 (0.0MB)
> free = 196608 (0.1875MB)
> 0.0% used
> tenured generation:
> capacity = 28311552 (27.0MB)
> used = 20464784 (19.516738891601562MB)
> free = 7846768 (7.4832611083984375MB)
> 72.28421811704283% used
> Perm Generation:
> capacity = 12582912 (12.0MB)
> used = 8834304 (8.425048828125MB)
> free = 3748608 (3.574951171875MB)
> 70.208740234375% used
>
> Here are my <Connector> configurations:
>
> <!-- Regular non-APR Coyote Connector -->
> <Connector port="8001"
> protocol="org.apache.coyote.http11.Http11Protocol"
> connectionTimeout="20000"
> server="Coyote1.1non-APR"
> />
>
> <!-- APR Connector -->
> <Connector port="8002"
> protocol="org.apache.coyote.http11.Http11AprProtocol"
> useSendfile="true"
> connectionTimeout="20000"
> server="Coyote1.1APR"
> />
>
> <!-- APR without sendfile -->
> <Connector port="8003"
> protocol="org.apache.coyote.http11.Http11AprProtocol"
> useSendfile="false"
> connectionTimeout="20000"
> server="Coyote1.1APRw/osendfile"
> />
>
> <!-- NIO Connector -->
> <Connector port="8004"
> protocol="org.apache.coyote.http11.Http11NioProtocol"
> useSendfile="true"
> connectionTimeout="20000"
> server="Coyote1.1NIO"
> />
>
> <!-- APR without sendfile -->
> <Connector port="8005"
> protocol="org.apache.coyote.http11.Http11NioProtocol"
> useSendfile="false"
> connectionTimeout="20000"
> server="Coyote1.1NIOw/osendfile"
> />
>
> All connectors are configured at once, so I should have a maximum of 40
> threads in each pool. The command I ran to benchmark each connector was
> (for example):
>
> /usr/sbin/ab -c 40 -t 480 -n 10000000 http://localhost:8004/4kiB.bin
>
> This runs ApacheBench for 8 minutes with 40 client threads requesting a
> 4k file over and over again. This particular test succeeded, but there
> are 14 more tests, each using a file twice the size of the previous
> test. After the 128k file test, every single test fails after that.
>
> The last test I ran (with only 1 thread instead of 40), the NIO
> connector died in the same way, but the NIO connector without sendfile
> enabled appeared to work properly. This time (40 threads), neither of
> the connectors worker properly, the NIO connector failing to complete
> any tests after the 128kb test and the NIO-sendfile connector failed to
> complete /all/ of the tests (automatically run immediately following the
> NIO tests).
>
> No OOMEs were encountered: only the exception shown above (no more
> files). On my previous tests, lsof reported that only one of my files
> was still open by the process. After this most recent test, it appears
> that 954 of my static files are still open by the process (and the test
> ended over 24 hours ago).
>
> The initial set of tests (c=1) seemed to recover, while this second set
> of tests (c=40) has not.
>
> My knee-jerk reaction is that the most number of files that should ever
> be open is 40: one per request processing thread. Something, somewhere
> is causing these file descriptors to stay open.
>
> Unfortunately, I don't have any GC information for the time period
> covering the test.
>
> I still have the JVM running, so I can probably inspect it for certain
> things if anyone has any questions. Unfortunately, I can't run any new
> JSP files (out of files!) and it looks like I can't connect using
> jconsole (probably because the JVM can't open a new socket).
>
> I'd love some suggestions at to what's going on, here.
Maybe looking at the directory /proc/PID/fd (sorry PID) will give more
info (at least of the FDs are still in use). Or you might have "lsof"
inspected. Sometimes proc is enough, sometimes you need lsof to find out
more.
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: NIO Connector: Too many open files [SOLVED]
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Filip,
On 5/22/2009 10:14 AM, Filip Hanik - Dev Lists wrote:
> [The NIO connector in 6.0.18] simply never called close on the FD when it was done
I tried the tomcat-coyote.jar from Tomcat 6.0.20 and everything worked
quite well. Thanks a lot for catching this problem and bringing it to my
attention.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkoX9VcACgkQ9CaO5/Lv0PBKagCgw/rZOhOKiuFIIQ5NFD+sQNRO
e1wAoJBERiOhKTaHFJeu/TqIjTQWGPWY
=3Vxj
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: NIO Connector: Too many open files
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Filip,
On 5/22/2009 10:14 AM, Filip Hanik - Dev Lists wrote:
> if you look at the commit, there are changelog changes in there. one of
> them being a FD leak with static content.
Gotcha. I was just looking at the commit comment itself.
> It simply never called close on the FD when it was done
Oops. Does this affect the connector regardless of the setting for
"sendFile"?
Also, I asked if using the trunk tcnative was the solution... of course
it's not: the bug was in the Java code. Should I get the tomcat-native
package and compile /that/ Java code, or should I get Tomcat 6.0.x trunk
and compile /that/?
I think I'd prefer to change as little as possible to resume my tests,
so if the connector itself is completely contained within the
tomcat-native package, and there won't be any incompatibilities with
6.0.18, I'd prefer to do that. In that case, do I just replace
lib/tomcat-coyote.jar after I build?
Thanks,
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: NIO Connector: Too many open files
Posted by Filip Hanik - Dev Lists <de...@hanik.com>.
Christopher Schultz wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Filip,
>
> On 5/21/2009 12:34 PM, Filip Hanik - Dev Lists wrote:
>
>> hi Christopher, generally, ulimit -n 1024 is too low for any kind of web
>> server.
>>
>
> Fair enough, but I'm not putting an unreasonable load on my server with
> ulimit -n 1024 and 40 concurrent connections, am I?
>
>
>> And there was also a file descriptor leak in the NIO connector, fixed in
>> http://svn.apache.org/viewvc?rev=734454&view=rev
>>
>> this is when Tomcat NIO serves up static content.
>>
>
> Yup, I'm only serving up static content (using sendfile in the first
> test that failed to complete, and NOT using sendfile in the second one
> which completed with c=1 but failed on c=40).
>
> Is tcnative trunk stable enough to just try using that?
>
> The commit comment says "NIO fixes"... nothing more specific? ;)
>
if you look at the commit, there are changelog changes in there. one of
them being a FD leak with static content.
It simply never called close on the FD when it was done
Filip
> Thanks,
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAkoWofMACgkQ9CaO5/Lv0PBGnACgjDwmquIeUv1FfeCYKQHEx2HZ
> s9AAn2UYoLK2J3exkh1Q0h8Wt8A0rJSa
> =8tQI
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: NIO Connector: Too many open files
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Filip,
On 5/21/2009 12:34 PM, Filip Hanik - Dev Lists wrote:
> hi Christopher, generally, ulimit -n 1024 is too low for any kind of web
> server.
Fair enough, but I'm not putting an unreasonable load on my server with
ulimit -n 1024 and 40 concurrent connections, am I?
> And there was also a file descriptor leak in the NIO connector, fixed in
> http://svn.apache.org/viewvc?rev=734454&view=rev
>
> this is when Tomcat NIO serves up static content.
Yup, I'm only serving up static content (using sendfile in the first
test that failed to complete, and NOT using sendfile in the second one
which completed with c=1 but failed on c=40).
Is tcnative trunk stable enough to just try using that?
The commit comment says "NIO fixes"... nothing more specific? ;)
Thanks,
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkoWofMACgkQ9CaO5/Lv0PBGnACgjDwmquIeUv1FfeCYKQHEx2HZ
s9AAn2UYoLK2J3exkh1Q0h8Wt8A0rJSa
=8tQI
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: NIO Connector: Too many open files
Posted by Filip Hanik - Dev Lists <de...@hanik.com>.
hi Christopher, generally, ulimit -n 1024 is too low for any kind of web
server.
And there was also a file descriptor leak in the NIO connector, fixed in
http://svn.apache.org/viewvc?rev=734454&view=rev
this is when Tomcat NIO serves up static content.
Filip
Christopher Schultz wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> All,
>
> I've been testing the performance of various Tomcat configurations
> against Apache httpd and my serious tests are not completing for the NIO
> connector because the server is running out of files:
>
>
>> May 20, 2009 2:35:55 AM org.apache.tomcat.util.net.NioEndpoint$Acceptor run
>> SEVERE: Socket accept failed
>> java.io.IOException: Too many open files
>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
>> at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
>> at java.lang.Thread.run(Thread.java:619)
>>
>
> A bit of background for those who haven't followed the "Apache httpd vs
> Tomcat static content performance" thread:
>
> I'm running Tomcat 6.0.18 using tcnative 1.1.16. Apache httpd is not
> being used for this test, so the client is contacting Tomcat directly
> from localhost.
>
> $ uname -a
> Linux chadis 2.6.14-gentoo-r5 #2 PREEMPT Sat Dec 17 16:30:55 EST 2005
> i686 AMD Athlon(tm) XP 1700+ AuthenticAMD GNU/Linux
>
> $ java -version
> java version "1.6.0_13"
> Java(TM) SE Runtime Environment (build 1.6.0_13-b03)
> Java HotSpot(TM) Client VM (build 11.3-b02, mixed mode, sharing)
>
> $ ulimit -n (fds per process limit)
> 1024
>
> 1GiB RAM on the machine, here are the heap details /after/ the tests
> are run:
>
> $ jmap -heap 1430
> Attaching to process ID 1430, please wait...
> Debugger attached successfully.
> Client compiler detected.
> JVM version is 11.3-b02
>
> using thread-local object allocation.
> Mark Sweep Compact GC
>
> Heap Configuration:
> MinHeapFreeRatio = 40
> MaxHeapFreeRatio = 70
> MaxHeapSize = 67108864 (64.0MB)
> NewSize = 1048576 (1.0MB)
> MaxNewSize = 4294901760 (4095.9375MB)
> OldSize = 4194304 (4.0MB)
> NewRatio = 12
> SurvivorRatio = 8
> PermSize = 12582912 (12.0MB)
> MaxPermSize = 67108864 (64.0MB)
>
> Heap Usage:
> New Generation (Eden + 1 Survivor Space):
> capacity = 2228224 (2.125MB)
> used = 612888 (0.5844955444335938MB)
> free = 1615336 (1.5405044555664062MB)
> 27.505672679227942% used
> Eden Space:
> capacity = 2031616 (1.9375MB)
> used = 612888 (0.5844955444335938MB)
> free = 1418728 (1.3530044555664062MB)
> 30.167511970766128% used
> - From Space:
> capacity = 196608 (0.1875MB)
> used = 0 (0.0MB)
> free = 196608 (0.1875MB)
> 0.0% used
> To Space:
> capacity = 196608 (0.1875MB)
> used = 0 (0.0MB)
> free = 196608 (0.1875MB)
> 0.0% used
> tenured generation:
> capacity = 28311552 (27.0MB)
> used = 20464784 (19.516738891601562MB)
> free = 7846768 (7.4832611083984375MB)
> 72.28421811704283% used
> Perm Generation:
> capacity = 12582912 (12.0MB)
> used = 8834304 (8.425048828125MB)
> free = 3748608 (3.574951171875MB)
> 70.208740234375% used
>
> Here are my <Connector> configurations:
>
> <!-- Regular non-APR Coyote Connector -->
> <Connector port="8001"
> protocol="org.apache.coyote.http11.Http11Protocol"
> connectionTimeout="20000"
> server="Coyote1.1non-APR"
> />
>
> <!-- APR Connector -->
> <Connector port="8002"
> protocol="org.apache.coyote.http11.Http11AprProtocol"
> useSendfile="true"
> connectionTimeout="20000"
> server="Coyote1.1APR"
> />
>
> <!-- APR without sendfile -->
> <Connector port="8003"
> protocol="org.apache.coyote.http11.Http11AprProtocol"
> useSendfile="false"
> connectionTimeout="20000"
> server="Coyote1.1APRw/osendfile"
> />
>
> <!-- NIO Connector -->
> <Connector port="8004"
> protocol="org.apache.coyote.http11.Http11NioProtocol"
> useSendfile="true"
> connectionTimeout="20000"
> server="Coyote1.1NIO"
> />
>
> <!-- APR without sendfile -->
> <Connector port="8005"
> protocol="org.apache.coyote.http11.Http11NioProtocol"
> useSendfile="false"
> connectionTimeout="20000"
> server="Coyote1.1NIOw/osendfile"
> />
>
> All connectors are configured at once, so I should have a maximum of 40
> threads in each pool. The command I ran to benchmark each connector was
> (for example):
>
> /usr/sbin/ab -c 40 -t 480 -n 10000000 http://localhost:8004/4kiB.bin
>
> This runs ApacheBench for 8 minutes with 40 client threads requesting a
> 4k file over and over again. This particular test succeeded, but there
> are 14 more tests, each using a file twice the size of the previous
> test. After the 128k file test, every single test fails after that.
>
> The last test I ran (with only 1 thread instead of 40), the NIO
> connector died in the same way, but the NIO connector without sendfile
> enabled appeared to work properly. This time (40 threads), neither of
> the connectors worker properly, the NIO connector failing to complete
> any tests after the 128kb test and the NIO-sendfile connector failed to
> complete /all/ of the tests (automatically run immediately following the
> NIO tests).
>
> No OOMEs were encountered: only the exception shown above (no more
> files). On my previous tests, lsof reported that only one of my files
> was still open by the process. After this most recent test, it appears
> that 954 of my static files are still open by the process (and the test
> ended over 24 hours ago).
>
> The initial set of tests (c=1) seemed to recover, while this second set
> of tests (c=40) has not.
>
> My knee-jerk reaction is that the most number of files that should ever
> be open is 40: one per request processing thread. Something, somewhere
> is causing these file descriptors to stay open.
>
> Unfortunately, I don't have any GC information for the time period
> covering the test.
>
> I still have the JVM running, so I can probably inspect it for certain
> things if anyone has any questions. Unfortunately, I can't run any new
> JSP files (out of files!) and it looks like I can't connect using
> jconsole (probably because the JVM can't open a new socket).
>
> I'd love some suggestions at to what's going on, here.
>
> Thanks,
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAkoVeX8ACgkQ9CaO5/Lv0PCN3QCfXV3tu/BIM7I8/dCvQ0ehkVws
> s/QAn3Kf4PjoAdSnUtp7FGMLz5u1W0FY
> =lz7Y
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org