You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomcat.apache.org by bu...@apache.org on 2012/10/22 15:52:15 UTC
[Bug 54036] New: currentThreadsBusy issue in centos 6
https://issues.apache.org/bugzilla/show_bug.cgi?id=54036
Priority: P2
Bug ID: 54036
Assignee: dev@tomcat.apache.org
Summary: currentThreadsBusy issue in centos 6
Severity: critical
Classification: Unclassified
OS: Linux
Reporter: vigotab@gmail.com
Hardware: PC
Status: NEW
Version: 7.0.32
Component: Connectors
Product: Tomcat 7
Hi, since when centos 6 went out I've tryed to switch some of my server from
centos 5.x to 6.x I currently own 13 servers with tomcat installed ,
thse servers have a very high load, and contain only 1 servlet never
reloaded/unloaded which print out static content (no cookie/session) i use the
compiled JNI (with apr ) , initially everythin was 1 servlet inside tomcat, now
i have directly embedded tomcat into the servlet,
the issue is:
when i have tomcat (jdk from 1.6.x to 1.7.x doesn't change) and the load
become a bit intense (about 600 req/sec ) the currentThreadsBusy become very
high till maxes out (220 currentThreadsBusy) and stay at that value, seems that
threads have some issue in closing ( i've inserted an atomic timer in servlets
to monitor if these remains up, and cuncurrent executions calculated by this
timer remain between 1 or 2 !!) and that's right because cpu usage is at 0.3%!
same identical configuration in a server with centos 5.x run fine with about 15
threads busy serving about 8k requests/second with cpu usage of about 35% (even
if cuncurrent servlet in execution are usually 1 or 2 here too), seems that
tomcat have some issue detaching the thread, i supposed that was my fault of
some configuration (even if i'm not so dumb on that ) the ni found that this
guy have the same problem :
http://efreedom.com/Question/1-7296648/Tomcat-Centos-CurrentThreadsBusy-Issue
no one seems to have a solution, initially i thought that was because centos 6
was pretty recent so maybe the issue would be fixed in earlyer releases of
tomcat, but we are at 7.0.32 and the problem persist..
when i have time i want to try to help in developement of tomcat (i'm learning
a bit of his internal syntax while working on it.. ) but at this moment i
don't even have time to sleep.. nor to provide some easily reproducible test
case , also because this need a bit of load from different sources, but if you
have suggestions i can try to implements and let you know if works , best i can
do now is to attach thread dump of the current bugged execution (with all
threads stuck here
at org.apache.tomcat.jni.Socket.recvbb(Native Method)
at
org.apache.coyote.http11.InternalAprInputBuffer.fill(InternalAprInputBuffer.java:575)
at
org.apache.coyote.http11.InternalAprInputBuffer.parseRequestLine(InternalAprInputBuffer.java:134)
please let me know.. at this moment i'm again going to reinstall centos 5.x on
that server and give up, because i need this server up and working today..
but anyway this issues must be fixed.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 54036] currentThreadsBusy issue in centos 6
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=54036
--- Comment #5 from francesco <vi...@gmail.com> ---
i understood sorry u meant that you have changed location of the report (i'm
not very used to send bug reports) no problem :)
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: [Bug 54036] currentThreadsBusy issue in centos 6
Posted by vigo tab <vi...@gmail.com>.
after further investigation seems that the problem is really at this point:
at org.apache.tomcat.jni.Socket.recvbb(Native Method)
at
org.apache.coyote.http11.InternalAprInputBuffer.fill(InternalAprInputBuffer.java:575)
each threads hang at this position till timeout and timeout on each
connections i just wonder why this doesn't happen in centos 5.x and instead
is bugged in centos 6.x probably some EOT/EOF signal didn't pass through
APR/JNI ?
2012/10/22 <bu...@apache.org>
> https://issues.apache.org/bugzilla/show_bug.cgi?id=54036
>
> --- Comment #1 from francesco <vi...@gmail.com> ---
> Created attachment 29506
> -->
> https://issues.apache.org/bugzilla/attachment.cgi?id=29506&action=edit
> thread dump
>
> thread dump of tomcat when providing the issue described ( incredibly high
> currentThreadsBusy ) which causes unusable tomcat installation
>
> --
> You are receiving this mail because:
> You are the assignee for the bug.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: dev-help@tomcat.apache.org
>
>
[Bug 54036] currentThreadsBusy issue in centos 6
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=54036
--- Comment #1 from francesco <vi...@gmail.com> ---
Created attachment 29506
--> https://issues.apache.org/bugzilla/attachment.cgi?id=29506&action=edit
thread dump
thread dump of tomcat when providing the issue described ( incredibly high
currentThreadsBusy ) which causes unusable tomcat installation
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 54036] currentThreadsBusy issue in centos 6
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=54036
francesco <vi...@gmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEEDINFO |NEW
--- Comment #9 from francesco <vi...@gmail.com> ---
i've used for bot centos 5 and 6,
tomcat-native-1.1.24
and
apr-1.4.6
*all tested environments are 64 bit
but as i've tested yesterday this issue happens also with bio connectors
the problem anyway exists since some months i have never got a working tomcat
in centos 6 (and maybe in fedora/RH lastest releases, i don't have tested there
but should be almost the same),
and because centos is one of the top used server distro i think that this
issue should be treated as critical
i think that people which try to use tomcat in centos into a server with a
quite high load, see the resource usage and think that tomcat is slow
which is not! they don't fill a bug report because they don't think is a but
(but instead it is because threads and connections doesn't close when tcp/ip
transmission end)
now i have reinstalled again centos 5.8 on that server (i'm stuck to centos 5
:( ) and the server is replying at about 4k connections / seconds with
8 cuncurrentthreadsBusy and 19 connectionCount (using APR) everything it's
exactly the same as previously tested in centos 6 (which was unable to go over
200 connections/seconds without beign slow because threads count would always
max out )
same sysctl.conf, firewall disabled , same tomcat version and same servlet
(servlet is compiled locally then rsync'ed )
i think that someone should take a look at this issue, if i have a bit of time
in next months i'll try to setup iptables to try to forward multiple packets to
virtual machine
with centos 5 and 6, simulating multiple ip connections (if u know a better way
to do that let me know ) anyway if someone is running tomcat on centos 6 with a
busy site
and don't have this problem tell there and you can say then that it's me that
i'm wrong,
elseway i'm pretty sure that there is a very important issue there -->
indipendent of servlet served <-- that should need attention :)
thank you,
Francesco
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 54036] currentThreadsBusy issue in centos 6
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=54036
Mark Thomas <ma...@apache.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |NEEDINFO
Component|Library |Connectors
Version|unspecified |7.0.32
Product|Tomcat Native |Tomcat 7
--- Comment #6 from Mark Thomas <ma...@apache.org> ---
If this occurs across multiple connectors then this is not an APR/native bug.
Moving back.
Please provide a reproducible test case. Without one, this will get closed as
INVALID.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 54036] currentThreadsBusy issue in centos 6
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=54036
--- Comment #13 from Brandon Chong <lo...@naver.com> ---
Hi, Francesco?
I have a similar problem as you reported above. I'm using CentOS 6.3 And Tomcat
6.0.35. If you figured out this problem already, could you let me know how to?
If you give me some advise I'll be very pleased!!! Please feel free to email me
at 'lovewill@naver.com'. Thank you in advance. :)
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 54036] currentThreadsBusy issue in centos 6
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=54036
--- Comment #4 from francesco <vi...@gmail.com> ---
Hi Mark do you mean to change connector?
i've tryed changing to BIO but the problem persist..
i have 220 threads blocked at this point:
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:150)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at
org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:516)
at
org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:501)
i have no other solution than change os for tomcat server, but anyway having
tomcat UNCOMPATIBLE (because this bug make the system very susceptible to even
small DDoS ) with the new CENTOS >= 6 is a BIG ISSUE i think.
, apache benchmark didn't show the issue in bot APR and BIO , ( keepalive is
off (value=1) but the problem show up only when lot of external ip connect)
maybe this is due on how the system handle tcp connections closing at lower
levels ?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 54036] currentThreadsBusy issue in centos 6
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=54036
--- Comment #2 from francesco <vi...@gmail.com> ---
after further investigation seems that the problem is really at this point:
at org.apache.tomcat.jni.Socket.recvbb(Native Method)
at
org.apache.coyote.http11.InternalAprInputBuffer.fill(InternalAprInputBuffer.java:575)
each threads hang at this position till timeout and timeout on each connections
i just wonder why this doesn't happen in centos 5.x and instead is bugged in
centos 6.x probably some EOT/EOF signal didn't pass through APR/JNI ?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 54036] currentThreadsBusy issue in centos 6
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=54036
Mark Thomas <ma...@apache.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Component|Connectors |Library
Version|7.0.32 |unspecified
Product|Tomcat 7 |Tomcat Native
--- Comment #3 from Mark Thomas <ma...@apache.org> ---
Changing component.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 54036] currentThreadsBusy issue in centos 6
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=54036
--- Comment #10 from francesco <vi...@gmail.com> ---
this error happened with java version since 1.6.xx to 1.7.xx last attempt of
yesterday has been done using the lastest 7u9-b05 , are months that i always
try with newer software available to get tomcat working correctly on centos 6
but nothing at the end i have always to giveup and reinstall centos 5 and with
same identical releases of software everything works correctly... the problem
here i think is that if you have never tryed with tomcat & centos 5 you think
that this is just a problem of performance but i can't believe that in centos 6
threads & connections remain opened this IS a bug, performance serving small
static content (to multiple destinations, apache benchmark tests doesn't show
this issue) are 100x worse in centos 6 and i mean 100x!! == 10 000% !
obviously if you have a huge servlet which take 1 second to serve just 1
request you may be not seeing this issue because the bottleneck is not so
evident so...
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 54036] currentThreadsBusy issue in centos 6
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=54036
Mark Thomas <ma...@apache.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution|--- |INVALID
--- Comment #12 from Mark Thomas <ma...@apache.org> ---
The thread dump shows that you have a large number of new connections that are
not sending any data. Hence, since BIO and APR/native both block during the
reading of the request line, the threads will remain blocked until the read
times out of the client sends more data.
You can see this by examining the stack trace. For example:
"http-apr-80-exec-96" - Thread t@137
java.lang.Thread.State: RUNNABLE
at org.apache.tomcat.jni.Socket.recvbb(Native Method)
at
org.apache.coyote.http11.InternalAprInputBuffer.fill(InternalAprInputBuffer.java:575)
at
org.apache.coyote.http11.InternalAprInputBuffer.parseRequestLine(InternalAprInputBuffer.java:134)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:927)
process(AbstractHttp11Processor.java:927)
This is when the processor triggers the read of the request line. The parameter
it passes in (keptAlive) indicates is this is the first request received on the
connection (keptAlive will be false) or a subsequent request (keptAlive will be
true). It is important for this issue to know if keptAlive is true or false.
parseRequestLine(InternalAprInputBuffer.java:134)
This is the reading data from the socket for the request line and parsing it.
Since we are at line 134 we know useAvailableData == false (else the if block
starting on line 131 would have been executed and the method would have
returned false to the caller. If useAvailableData == false then keptAlive ==
false. This tells us that this is the first request on the connection.
fill(InternalAprInputBuffer.java:575)
This is the InputBuffer attempting to read enough (actually any) data into the
buffer so it can be parsed as a request line. It is blocking as there is no
data to read.
Putting everything together we know that:
- this thread is processing a new connection from a client
- the connection has been established but no data has been sent
- Tomcat is waiting for data to arrive so it can parse the request line
If you have lots of clients doing this then you have some broken clients.
Switching to the NIO connector may help since it is non-blocking while reading
the request line and request headers (all connectors block while reading
request bodies).
I have no idea what it is between your CentOS 5 and CentOS 6 installations that
is triggering this issue. It may be related to defer accept but still, the
clients are misbehaving.
There is no Tomcat bug here. If you need further assistance, the Tomcat users
mailing list is the place to seek help.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 54036] currentThreadsBusy issue in centos 6
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=54036
--- Comment #11 from Pid <bu...@pidster.com> ---
Creating a load test that sends 100s of requests per second isn't so hard.
A test case that reproduces a particular outcome is the only way for the Tomcat
community to analyse and fix an issue.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 54036] currentThreadsBusy issue in centos 6
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=54036
--- Comment #7 from francesco <vi...@gmail.com> ---
you are the good tomcat programmer , how i can write a reproducible case if you
cannot reproduce the volume of requests i'm getting (these are TCP connections
not udp so i don't know how these can be simulated as coming from multiple
different sources (maybe with something like playing with NAT in iptables) ) i
am not the only one who has this issue, is someone use tomcat live with centos
6 and have more than 100 visit /seconds should have faced this issue too... the
test case doesn't need to be built because happens always on centos >= 6
(and probably on RHEL but i haven't tested before) , threads didn't close at
the end of connections but at the socket timeout, so if you have high
connection rate from different sources the thread pool become exhausted soon..
if you know how to reproduce these multiple connections i'm almost sure that
also the HelloWorld servlet would works as test case in centos 6
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 54036] currentThreadsBusy issue in centos 6
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=54036
--- Comment #8 from Pid <bu...@pidster.com> ---
Which versions of APR and Java are installed on the Centos 5 and 6 systems?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org