You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Barak Yaish <ba...@gmail.com> on 2009/02/02 10:53:36 UTC
Tomcat resets the connection during load
I'm running a Tomcat 6.0.16 (native library is compiled )on a Linux box
(CentOS 5), java version is:
java version "1.6.0"
Java(TM) SE Runtime Environment (build 1.6.0-b105)
Java HotSpot(TM) Server VM (build 1.6.0-b105, mixed mode)
The connection configuration is:
<Connector port="8080" protocol="org.apache.coyote.http11.Http11AprProtocol"
connectionTimeout="10000" enableLookups="false"
maxKeepAliveRequests="1" maxThreads="500" minSpareThreads="5"
maxSpareThreads="40" acceptCount="1000" />
I've deployed my wepapp into tomcat, which on low load (few requests per
second) the doGet() method lasts about 30-40ms.
In order to measure the performance of the webapp under load, I wrote a
simple test that simulates high load on tomcat (hundreds of requests per
second). After few seconds of running, the client received the following
exceptions from the server (not all the requests, some of them processed and
return response, but it took few seconds though):
java.net.SocketException: Unexpected end of file from server
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:714)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:577)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:711)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:577)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1004)
at AdvertiserCaller.call(StressTest.java:192)
at AdvertiserCaller.call(StressTest.java:166)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
I've records the traffic using Wireshark (http://www.wireshark.org), and
I've noticed that Tomcat send TCP Reset to the client, before it writes the
reponse to the stream.
Can you please assist understanding this behavior and tuning Tomcat in order
to avoid these problems?
Thanks,
Barak.
Re: Tomcat resets the connection during load
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Mark,
Mark Thomas wrote:
> There is a proposal to log a message when [the thread pool is
> exhausted and accept queue is full] but with current versions you
> won't see anything in the logs.
Can Java even detect the second of those two conditions? I would think
the best you could do is make a note that the last available request
processor is being taken out of the pool to service a request.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkmLacgACgkQ9CaO5/Lv0PCIFQCgpx7QWxqyrpqmVEIOnpfiPD6d
knkAnArm/6Vx2IgsmrlsYMkun+Gw6loa
=LKPe
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat resets the connection during load
Posted by Barak Yaish <ba...@gmail.com>.
Ok, will try to search after the relevant attributes in jconsole, for the
mean time I ran ab against a load balancer connected to 3 Tomcat on 3
different boxes and the scenario repeated it self:
[admin@ltse1 ~]$ ab -n 20000 -k -c 1500 "
http://lbmachine/scale/ctx1?cc=123456"
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/
Benchmarking *lbmachine* (be patient)
Completed 2000 requests
Completed 4000 requests
Completed 6000 requests
apr_socket_recv: Connection reset by peer (104)
Total of 6576 requests completed
On Tue, Feb 3, 2009 at 3:48 PM, Mark Thomas <ma...@apache.org> wrote:
> Barak Yaish wrote:
> > How can I determine that? I haven't seen any exception in catalina.out
>
> Monitor the current thread usage (eg with LambdaProbe, JMC console, manager
> app)
>
> > regarding out of threads...
>
> There is a proposal to log a message when this happens but with current
> versions
> you won't see anything in the logs.
>
> Mark
>
> >
> > On Tue, Feb 3, 2009 at 6:35 AM, Mark Thomas <ma...@apache.org> wrote:
> >
> >> Barak Yaish wrote:
> >>
> >>> I've records the traffic using Wireshark (http://www.wireshark.org),
> and
> >>> I've noticed that Tomcat send TCP Reset to the client, before it writes
> >> the
> >>> reponse to the stream.
> >>>
> >>> Can you please assist understanding this behavior and tuning Tomcat in
> >> order
> >>> to avoid these problems?
> >> Did you run out of threads and fill the accept queue?
> >>
> >> Mark
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> For additional commands, e-mail: users-help@tomcat.apache.org
> >>
> >>
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Tomcat resets the connection during load
Posted by Mark Thomas <ma...@apache.org>.
Barak Yaish wrote:
> How can I determine that? I haven't seen any exception in catalina.out
Monitor the current thread usage (eg with LambdaProbe, JMC console, manager app)
> regarding out of threads...
There is a proposal to log a message when this happens but with current versions
you won't see anything in the logs.
Mark
>
> On Tue, Feb 3, 2009 at 6:35 AM, Mark Thomas <ma...@apache.org> wrote:
>
>> Barak Yaish wrote:
>>
>>> I've records the traffic using Wireshark (http://www.wireshark.org), and
>>> I've noticed that Tomcat send TCP Reset to the client, before it writes
>> the
>>> reponse to the stream.
>>>
>>> Can you please assist understanding this behavior and tuning Tomcat in
>> order
>>> to avoid these problems?
>> Did you run out of threads and fill the accept queue?
>>
>> Mark
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat resets the connection during load
Posted by Barak Yaish <ba...@gmail.com>.
How can I determine that? I haven't seen any exception in catalina.out
regarding out of threads...
On Tue, Feb 3, 2009 at 6:35 AM, Mark Thomas <ma...@apache.org> wrote:
> Barak Yaish wrote:
>
> > I've records the traffic using Wireshark (http://www.wireshark.org), and
> > I've noticed that Tomcat send TCP Reset to the client, before it writes
> the
> > reponse to the stream.
> >
> > Can you please assist understanding this behavior and tuning Tomcat in
> order
> > to avoid these problems?
>
> Did you run out of threads and fill the accept queue?
>
> Mark
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Tomcat resets the connection during load
Posted by Mark Thomas <ma...@apache.org>.
Barak Yaish wrote:
> I've records the traffic using Wireshark (http://www.wireshark.org), and
> I've noticed that Tomcat send TCP Reset to the client, before it writes the
> reponse to the stream.
>
> Can you please assist understanding this behavior and tuning Tomcat in order
> to avoid these problems?
Did you run out of threads and fill the accept queue?
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org