You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomcat.apache.org by Jess Holle <je...@ptc.com> on 2007/06/14 23:58:20 UTC
Serious non-native AJP connector issue
We're facing a /serious /issue with the non-native AJP connector.
To put it most simply, some requests seem to "get lost" in Tomcat in
various cases involving concurrent requests -- and not egregious numbers
of concurrent requests, either.
For instance,
1. Use a Tomcat 5.5.23 with a configuration like:
<Connector port="8010"
minSpareThreads="4" maxSpareThreads="12"
maxThreads="18" acceptCount="282"
tomcatAuthentication="false"
useBodyEncodingForURI="true" URIEncoding="UTF-8"
enableLookups="false" redirectPort="8443"
protocol="AJP/1.3" />
(which are intended solely for making it easier to test concurrency
issues that to overrun a realistic 'maxThreads' setting) and as a
control, similar thread pool settings on the direct HTTP connector:
<Connector port="8080" maxHttpHeaderSize="8192"
minSpareThreads="4" maxSpareThreads="12"
maxThreads="18" acceptCount="282"
enableLookups="false" redirectPort="8443"
connectionTimeout="20000" disableUploadTimeout="true" />
2. Use an Apache 2.2.4 with mod_proxy_ajp with a configuration like:
<Proxy balancer://ajpWorker>
BalancerMember ajp://localhost:8010 min=16 max=300 smax=40
ttl=900 keepalive=Off timeout=900
</Proxy>
RewriteEngine on
RewriteRule ^(/TestApp/(.*\.jsp(.*)|servlet/.*|.*\.jar))$
balancer://ajpWorker$1 [P]
(on Windows in this case; similar results can be obtained on Linux
at least)
3. Use a simple test JSP page (placed in a web app containing nothing
else):
<%@page session="false"
%><%@page contentType="text/html" pageEncoding="UTF-8"
%><%!
private static final String titleString = "Sleepy Test JSP Page";
%><html>
<head>
<%
String sleepSeconds = request.getParameter( "secs" );
if ( sleepSeconds == null )
sleepSeconds = "1";
long secsToSleep = Long.parseLong( sleepSeconds );
Thread.sleep( 1000L * secsToSleep );
%>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title><%=titleString%></title>
</head>
<body>
<center>
<h2><%=titleString%>: SUCCESS!</h2>
[Slept <%= secsToSleep %> seconds.]
</center>
</body>
</html>
4. Hit the page with ab
* First, test direct Tomcat connections:
o ab -n 24 -c 24 -A wcadmin:wcadmin
http://hostname:*8080*/TestApp/test.jsp?secs=3
+ Result: All requests complete sucessfully in
6118 ms.
* Second, test connections via Apache:
o ab -n 24 -c 24 -A wcadmin:wcadmin
http://hostname/TestApp/test.jsp?secs=3
+ Result: Only 17 requests complete before ab
times out.
* Third, test connections via Apache again soon (under the
BalancerMember 'timeout' seconds) after the last test
o ab -n 24 -c 24 -A wcadmin:wcadmin
http://hostname/TestApp/test.jsp?secs=3
+ Result: Only 9 requests complete before ab times
out.
Something is clearly /horribly/ wrong with the handling of any
concurrency over 'maxThreads' in this case. Even so, there seems to be
some sort of "off-by-one" error in that only 17 requests complete, not
18. Worse, this has a lingering effect that decreases Tomcat's ability
to concurrent requests thereafter (with this impact seemingly being much
worse the longer the BalancerMember timeout is set to -- and we have
some very long running requests and thus need this timeout to be /very/
large).
This is not the only ill effect we've seen when hitting Tomcat 5.5.24
with concurrent requests in this manner, but it is a good place to start.
As for the native connector, it just works here. So why don't I just
use it? Well, first off, we have to support Tomcat on Windows (32 and
64-bit), Linux, Solaris, HPUX (PA-RISC and Itanium), and AIX. We've
been unable to get the connector built on some of these platforms and on
some we can't get the resulting binary to function (specifically on
AIX). Further, we had some stability issues with the native connector
in the past and had considered the Java connector the safest, if not
fastest, option -- and to a degree given that everything is Java I still
feel that's the case. Finally, however, this connector should just
plain work. Tomcat shouldn't be a cripple unless/until you manage to
build a native connector for your platform.
Any troubleshooting and/or debugging ideas (e.g. where exactly to place
breakpoints, what logs to turn on, etc, etc) would be /greatly/ appreciated.
--
Jess Holle
jessh@ptc.com
P.S. If this should go to the user's mailing list instead that's fine,
but this really seemed like a developer-oriented issue to me.
Re: Serious non-native AJP connector issue
Posted by Jess Holle <je...@ptc.com>.
Rainer Jung wrote:
> Jess,
>
> I didn't really carefully think about the case of a saturated pool.
> But nevertheless some hints:
>
> 1) There is always one thread waiting for the accept, which is a usual
> pool thread. So an offset of one between threeads processing requests
> and the pool size is normal.
Got that, but I've accounted for that via maxThreads of 51 and
BalanceMember max of 50.
I'm left wondering why there's an off-by-one beyond this (and thus an
off-by-2 overall).
> 2) There is no orderly shutdown in AJP13 for a connection, neither
> from the httpd side, not from the Tomcat side. mod_jk will realize
> closed connections and reopen them, but I have the impression, that
> mod_proxy fails a backend when it gets a connection error.
>
> Tomcat in my experience reliably closes a connection and frees the
> thread, when the web server does.
>
> Most of the time it makes sense to have a connectionTimeout
> (Milliseconds) and a connection_pool_timeout (seconds, mod_jk) resp.
> ttl (seconds, mod_proxy) which are in sync. But: they will never
> harmonize completely, because mod_jk only checks for the need to throw
> away closed connections during maintenance (every 60 seconds or
> whatever is configured with worker.maintain) and I think mod_proxy
> checks the ttl whenever a connection is put back into the pool.
I don't think any of those should be involved in this short test.
> 3) Does it happen with mod_jk too?
I believe so. I did some mod_jk testing to verify the overall nature of
the issue remained the same, but I didn't go through all the same
detailed tests. I could redo this particular test if that's helpful.
> 4) Weird guesses:
>
> - max is limited with mod_proxy to the number of threads per process
> configured in your MPM (worker?).
>
> This is 25 by default. So if we want to have an easy to understand
> scenario, set your threads per process to say 60 and min=max=50 as
> well as removing the smax and the ttl.. That way 50 connections should
> be started on startup (per httpd process; check with netstat), and we
> shouldn't see any resizing during your ab test. Now start your ab test
> before Tomcat will timeout (I remember that connectionTimeout had some
> default value, even if you don't set it, but it is in the minutes).
My Apache MaxClients is 300 and this is on Windows so I only have 1
worker process.
> If you don't run into trouble then, we know your observation has to do
> with resizing of connection pools.
>
> *Maybe*: ab is too fast and can come back with new requests faster,
> than the connections go back into the pool after the end of a request
> in httpd. Not very reasonable, but possible. Of cource the pool is
> synchronized and the lock doesn't need to behave fair, i.e. when it
> gets contended, it's not clear if the oldest waiting thread gets it
> first.
I believe I disproved this at some point by running 2 tests with -n 50
and -c 50 manually back to back, but I'd have to re-test to be sure.
[I'm wishing I'd taken better notes of various results...]
Apart from this weird edge condition (an off-by-2 isn't that devastating
if it stays "2" in all cases), the thing that gets me overall is that
the documentation really makes it sound like "acceptCount" works like a
fair queue and that there's no harm in exceeding maxThreads except that
requests will be queued. As Bill suggested, I should come up with
suggested patches to the documentation -- I'm just not yet confident
enough in my understanding to propose such patches. At this point all I
can propose is strong warning verbage around maxThreads and acceptCount
regarding their behavior for the Java AJP connector.
--
Jess Holle
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Serious non-native AJP connector issue
Posted by Rainer Jung <ra...@kippdata.de>.
Jess,
I didn't really carefully think about the case of a saturated pool. But
nevertheless some hints:
1) There is always one thread waiting for the accept, which is a usual
pool thread. So an offset of one between threeads processing requests
and the pool size is normal.
2) There is no orderly shutdown in AJP13 for a connection, neither from
the httpd side, not from the Tomcat side. mod_jk will realize closed
connections and reopen them, but I have the impression, that mod_proxy
fails a backend when it gets a connection error.
Tomcat in my experience reliably closes a connection and frees the
thread, when the web server does.
Most of the time it makes sense to have a connectionTimeout
(Milliseconds) and a connection_pool_timeout (seconds, mod_jk) resp. ttl
(seconds, mod_proxy) which are in sync. But: they will never harmonize
completely, because mod_jk only checks for the need to throw away closed
connections during maintenance (every 60 seconds or whatever is
configured with worker.maintain) and I think mod_proxy checks the ttl
whenever a connection is put back into the pool.
3) Does it happen with mod_jk too?
4) Weird guesses:
- max is limited with mod_proxy to the number of threads per process
configured in your MPM (worker?).
This is 25 by default. So if we want to have an easy to understand
scenario, set your threads per process to say 60 and min=max=50 as well
as removing the smax and the ttl.. That way 50 connections should be
started on startup (per httpd process; check with netstat), and we
shouldn't see any resizing during your ab test. Now start your ab test
before Tomcat will timeout (I remember that connectionTimeout had some
default value, even if you don't set it, but it is in the minutes).
If you don't run into trouble then, we know your observation has to do
with resizing of connection pools.
*Maybe*: ab is too fast and can come back with new requests faster, than
the connections go back into the pool after the end of a request in
httpd. Not very reasonable, but possible. Of cource the pool is
synchronized and the lock doesn't need to behave fair, i.e. when it gets
contended, it's not clear if the oldest waiting thread gets it first.
Regards,
Rainer
Jess Holle wrote:
> Hmmmm....
>
> I just redid my tests with:
>
> BalancerMember ajp://localhost:8010 min=15 max=*50* smax=30 ttl=900
> keepalive=Off timeout=900
>
> and
>
> <Connector port="8010"
> minSpareThreads="15" maxSpareThreads="30"
> maxThreads="*51*" acceptCount="*0*"
> tomcatAuthentication="false"
> useBodyEncodingForURI="true" URIEncoding="UTF-8"
> enableLookups="false" redirectPort="8443"
> protocol="AJP/1.3" />
>
> and
>
> ab -n 100 -c *50* http://jessh03l.ptcnet.ptc.com/TestApp/test.jsp?secs=3
>
> I (after about 3 seconds) get
>
> SEVERE: All threads (51) are currently busy, waiting. Increase
> maxThreads (51) or check the servlet status
>
> and I eventually get exactly 1 lost request. I'm baffled as to why this
> can occur.
>
> Something's still doesn't seem quite right here.
>
> What's even weirder is that I only get this issue with the first ab run
> after restarting Tomcat. If I do the same test again (any number of
> times) I don't lose any requests.
>
> I can get the same result by restarting and doing 2 ab runs with "-n
> 100" in fairly short secession, so this isn't some ab error. By "fairly
> short", I don't mean very short -- I left a good 5 seconds between runs.
>
> I find that using a max of 49 in Apache seems to work reliably, but I'm
> struggling to understand: (1) why I have to have 2 more Tomcat connector
> threads allowed than the number of connections I can possibly have and
> (2) whether "2" is always the safe buffer or whether I'll need 9 on an 8
> CPU box or some such.
>
> --
> Jess Holle
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Serious non-native AJP connector issue
Posted by Jess Holle <je...@ptc.com>.
Hmmmm....
I just redid my tests with:
BalancerMember ajp://localhost:8010 min=15 max=*50* smax=30 ttl=900
keepalive=Off timeout=900
and
<Connector port="8010"
minSpareThreads="15" maxSpareThreads="30"
maxThreads="*51*" acceptCount="*0*"
tomcatAuthentication="false"
useBodyEncodingForURI="true" URIEncoding="UTF-8"
enableLookups="false" redirectPort="8443"
protocol="AJP/1.3" />
and
ab -n 100 -c *50* http://jessh03l.ptcnet.ptc.com/TestApp/test.jsp?secs=3
I (after about 3 seconds) get
SEVERE: All threads (51) are currently busy, waiting. Increase
maxThreads (51) or check the servlet status
and I eventually get exactly 1 lost request. I'm baffled as to why this
can occur.
Something's still doesn't seem quite right here.
What's even weirder is that I only get this issue with the first ab run
after restarting Tomcat. If I do the same test again (any number of
times) I don't lose any requests.
I can get the same result by restarting and doing 2 ab runs with "-n
100" in fairly short secession, so this isn't some ab error. By "fairly
short", I don't mean very short -- I left a good 5 seconds between runs.
I find that using a max of 49 in Apache seems to work reliably, but I'm
struggling to understand: (1) why I have to have 2 more Tomcat connector
threads allowed than the number of connections I can possibly have and
(2) whether "2" is always the safe buffer or whether I'll need 9 on an 8
CPU box or some such.
--
Jess Holle
Re: Serious non-native AJP connector issue
Posted by Jess Holle <je...@ptc.com>.
Bill Barker wrote:
> It seems that we lost building the NIO/AJP connector in Tomcat, but since i
> recognize you from dev@httpd, you will have seen that already :). Last time
> I tested, the experimental NIO/AJP connector performed better than either
> the default or the APR connector on Solaris (surprise, surprise), and came
> in dead last on Windows.
>
Thanks for all the helpful information, Bill.
I wonder if the results of for the NIO/AJP connector would be any
different on Windows with a later JVM (e.g. Java 6 Update 1). I would
*hope* that Sun would fix the NIO issues on Windows to the point that it
would out-perform Java-Blocking-IO/AJP.
--
Jess Holle
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Serious non-native AJP connector issue
Posted by Bill Barker <wb...@wilshire.com>.
"Jess Holle" <je...@ptc.com> wrote in message news:4671FC01.60408@ptc.com...
> Okay that's all starting to make some sense, but it hard to see how
> someone would come to this understanding from the documentation.
>
> I also don't see how "|connectionTimeout" will help here in that the
> documentation says:
> |
>
> The number of milliseconds this *Connector* will wait, after
> accepting a connection, for the request URI line to be presented.
> The default value is infinite (i.e. no timeout).
>
In practice, this is a timeout on a wait on a keep-alive connection (since
request-body processing usually happens fast). If this particular
connection in the pool isn't reused in the time allowed, then Tomcat will
close it and put the thread back in the queue to process new connections.
Using CPing/CPong is recommended in this case to allow httpd to understand
that Tomcat has hung up the phone. So, if after the first request, httpd
chooses to not reuse this connection for a long time, then the thread can
recycle itself.
> If a connection is formed between Apache and Tomcat for 24 requests and
> 17 are immediately processed due to a maxThreads limit of 18, how would
> setting a low "connectionTimeout" help the 17 threads process the other
> 7 requests?
>
It would cause the TC threads listening for a new request to believe that it
will never come, and they will throw themselves back into the pool.
> I'm trying to understand if there is any useful throttling configuration
> in which the Java AJP connector has a maxThreads less than Apache's max
> AJP connections -- plus 1. Put another way, I'm not quite understanding
> any useful "acceptCount" scenario from the "connectionTimeout"
> description above.
>
Using acceptCount is normally useless. The case where it is is when you
suddenly get a very very large number of hits to your app faster than Tomcat
can handle.
> I know the Java AJP connector uses a thread-per-connection model. I had
> assumed that the "maxThreads" meant maximum /active request processing
> /threads (e.g. like some old Tomcat releases used to call this
> "maxProcessors") and that connections covered by "acceptCount" were
> still allocated threads and were still accepted (as the parameter name
> implies). I'd further assumed that a fair blocking queue arrangement
> would allow "maxThreads" connection threads to run and keep the others
> waiting until their turn. I don't see anything in the documentation to
> the contrary of this. Further I'm trying to understand any sort of
> arrangement with "connectionTimeout" that would give this level of
> utility to "acceptCount". As it stands it seems like users can easily
> produce devastatingly bad behavior by making assumptions about
> "acceptCount" that seem quite logical from the documentation.
>
The acceptCount is only useful to handle unexpected floods of requests.
Using the connectionTimeout means that the Tomcat thread won't just sit and
listen forever for that particler socket connection to send another request,
and will be freed up to be re-assigned to another socket connection. With
any reasonable setting, the connectionTimeout is just how long Tomcat will
maintain a keep-alive with httpd (which is very different from how long
httpd will keep a keep-alive with the client).
> Obviously the APR connector can do much better by /not/ allocating
> threads to connections beyond "maxThreads". That's great, but juggling
> native builds for many different OS's can be a real issue (which is why
> I still hold out hope that the NIO connection will come through with
> something better than the non-NIO Java connector's performance even if
> it is not APR-level performance).
>
It seems that we lost building the NIO/AJP connector in Tomcat, but since i
recognize you from dev@httpd, you will have seen that already :). Last time
I tested, the experimental NIO/AJP connector performed better than either
the default or the APR connector on Solaris (surprise, surprise), and came
in dead last on Windows.
> I don't mean to be argumentative -- I'm just really struggling to
> understand and hopefully preventing future misunderstandings through
> more clarity in the documentation.
>
With the documentation, as with the code, patches are always welcome :).
> --
> Jess Holle
>
> Bill Barker wrote:
>> "Jess Holle" <je...@ptc.com> wrote in message
>> news:4671E278.8020204@ptc.com...
>>
>>> There's no intent to handle this case? Really?
>>>
>>> Tomcat should throw an "IllegalArgumentException" whenever 'acceptCount'
>>> is set to anything other than 0 explaining this if this is the case!
>>>
>>> If so, this is very unfortunate. We use "maxThreads" as a throttle to
>>> limit the concurrency at this level under the (silly?) assumption that
>>> "acceptCount" behaves as documented in the documentation. [Yes, one
>>> could argue that a separate throttle should be used behind this layer,
>>> which we might have been inclined to do if the documentation said
>>> "acceptCount does not work".]
>>>
>>>
>>
>> You are misunderstanding how AJP works. Since you don't have a
>> connectionTimeout on the <Connector />, the connections to httpd stay
>> alive
>> waiting to get another request on the same socket. As a result, there
>> won't
>> be any free Threads to handle a new connection so it doesn't matter what
>> acceptCount is. Their isn't anyone there to accept them.
>>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Serious non-native AJP connector issue
Posted by Jess Holle <je...@ptc.com>.
Okay that's all starting to make some sense, but it hard to see how
someone would come to this understanding from the documentation.
I also don't see how "|connectionTimeout" will help here in that the
documentation says:
|
The number of milliseconds this *Connector* will wait, after
accepting a connection, for the request URI line to be presented.
The default value is infinite (i.e. no timeout).
If a connection is formed between Apache and Tomcat for 24 requests and
17 are immediately processed due to a maxThreads limit of 18, how would
setting a low "connectionTimeout" help the 17 threads process the other
7 requests?
I'm trying to understand if there is any useful throttling configuration
in which the Java AJP connector has a maxThreads less than Apache's max
AJP connections -- plus 1. Put another way, I'm not quite understanding
any useful "acceptCount" scenario from the "connectionTimeout"
description above.
I know the Java AJP connector uses a thread-per-connection model. I had
assumed that the "maxThreads" meant maximum /active request processing
/threads (e.g. like some old Tomcat releases used to call this
"maxProcessors") and that connections covered by "acceptCount" were
still allocated threads and were still accepted (as the parameter name
implies). I'd further assumed that a fair blocking queue arrangement
would allow "maxThreads" connection threads to run and keep the others
waiting until their turn. I don't see anything in the documentation to
the contrary of this. Further I'm trying to understand any sort of
arrangement with "connectionTimeout" that would give this level of
utility to "acceptCount". As it stands it seems like users can easily
produce devastatingly bad behavior by making assumptions about
"acceptCount" that seem quite logical from the documentation.
Obviously the APR connector can do much better by /not/ allocating
threads to connections beyond "maxThreads". That's great, but juggling
native builds for many different OS's can be a real issue (which is why
I still hold out hope that the NIO connection will come through with
something better than the non-NIO Java connector's performance even if
it is not APR-level performance).
I don't mean to be argumentative -- I'm just really struggling to
understand and hopefully preventing future misunderstandings through
more clarity in the documentation.
--
Jess Holle
Bill Barker wrote:
> "Jess Holle" <je...@ptc.com> wrote in message
> news:4671E278.8020204@ptc.com...
>
>> There's no intent to handle this case? Really?
>>
>> Tomcat should throw an "IllegalArgumentException" whenever 'acceptCount'
>> is set to anything other than 0 explaining this if this is the case!
>>
>> If so, this is very unfortunate. We use "maxThreads" as a throttle to
>> limit the concurrency at this level under the (silly?) assumption that
>> "acceptCount" behaves as documented in the documentation. [Yes, one
>> could argue that a separate throttle should be used behind this layer,
>> which we might have been inclined to do if the documentation said
>> "acceptCount does not work".]
>>
>>
>
> You are misunderstanding how AJP works. Since you don't have a
> connectionTimeout on the <Connector />, the connections to httpd stay alive
> waiting to get another request on the same socket. As a result, there won't
> be any free Threads to handle a new connection so it doesn't matter what
> acceptCount is. Their isn't anyone there to accept them.
>
Re: Serious non-native AJP connector issue
Posted by Bill Barker <wb...@wilshire.com>.
"Jess Holle" <je...@ptc.com> wrote in message
news:4671E278.8020204@ptc.com...
> There's no intent to handle this case? Really?
>
> Tomcat should throw an "IllegalArgumentException" whenever 'acceptCount'
> is set to anything other than 0 explaining this if this is the case!
>
> If so, this is very unfortunate. We use "maxThreads" as a throttle to
> limit the concurrency at this level under the (silly?) assumption that
> "acceptCount" behaves as documented in the documentation. [Yes, one
> could argue that a separate throttle should be used behind this layer,
> which we might have been inclined to do if the documentation said
> "acceptCount does not work".]
>
You are misunderstanding how AJP works. Since you don't have a
connectionTimeout on the <Connector />, the connections to httpd stay alive
waiting to get another request on the same socket. As a result, there won't
be any free Threads to handle a new connection so it doesn't matter what
acceptCount is. Their isn't anyone there to accept them.
> Further, I've been able to reproduce such issues using a concurrency of
> exactly the "maxThreads". At *best* there is an off-by-one error here,
> but I've still seen problems in some testing using a concurrency level 2
> or 3 less than "maxThreads" -- which leads me to believe there is
> something a bit more nefarious going on here. Another of my colleagues
> claims that he's encountered issues in simple ab tests quite a ways
> below "maxThreads". I've not managed to reproduce this case myself, but
> I could certainly focus on this case if the Tomcat contributors have no
> intent to fix "acceptCount" behavior for the Java AJP connector.
>
The thread waiting to accept new connections counts as a thread for the
purposes of maxThreads, so yes it is one less. And, I can save you the
trouble of investigating, since as long as your maxThreads <= pool size in
httpd you will see this sort of thing.
> Is there a firm intent to handle this case in the _/native/_ AJP
> connector? If so, then this distinction should be clearly stated (and
> an IllegalArgumentException thrown as per above when the native
> connector cannot be found/initialized).
>
The APR/AJP connector (as well as the experimental NIO/AJP connector) use a
many-to-one connection-to-thread setup so you can have a large number of
connections sitting and doing nothing and use relatively few threads. With
the default Java connector the mapping is one-to-one.
> Personally I see utility in having "acceptCount" working in the AJP
> case. If one exceeds the mod_proxy_ajp hard maximum the client
> immediately gets a 503. There are cases when just queuing a few
> requests makes more sense than either returning 503's or letting them
> rip -- which is where acceptCount comes in. [Granted the acceptCount
> numbers below are excessive, but that's another matter...]
>
It's not that acceptCount doesn't work, it's that without a
connectionTimeout each thread waits forever listening on it's connection for
the next request to come in. With a connectionTimeout, once httpd hasn't
sent another request on this connection after the timeout, the thread is
recycled and can handle a new connection.
> --
> Jess Holle
> jessh@ptc.com
>
> Bill Barker wrote:
>> Yes, since you know in advance how many connections you are going to get
>> to
>> the AJP connector, you can configure it so that it has enough threads to
>> handle all of those connections. That is why it doesn't attempt to
>> handle
>> the case when the concurrency goes above maxThreads
>>> -----Original Message-----
>>> From: Jess Holle [mailto:jessh@ptc.com]
>>> Sent: Thursday, June 14, 2007 2:58 PM
>>> To: Tomcat Developers List
>>> Cc: Dobson, Simon; Wang, Andy; Fenlason, Josh
>>> Subject: Serious non-native AJP connector issue
>>>
>>> We're facing a /serious /issue with the non-native AJP connector.
>>>
>>> To put it most simply, some requests seem to "get lost" in Tomcat in
>>> various cases involving concurrent requests -- and not
>>> egregious numbers
>>> of concurrent requests, either.
>>>
>>> For instance,
>>>
>>> 1. Use a Tomcat 5.5.23 with a configuration like:
>>>
>>> <Connector port="8010"
>>> minSpareThreads="4" maxSpareThreads="12"
>>> maxThreads="18" acceptCount="282"
>>> tomcatAuthentication="false"
>>> useBodyEncodingForURI="true" URIEncoding="UTF-8"
>>> enableLookups="false" redirectPort="8443"
>>> protocol="AJP/1.3" />
>>>
>>> (which are intended solely for making it easier to test
>>> concurrency
>>> issues that to overrun a realistic 'maxThreads' setting) and as a
>>> control, similar thread pool settings on the direct HTTP
>>> connector:
>>>
>>> <Connector port="8080" maxHttpHeaderSize="8192"
>>> minSpareThreads="4" maxSpareThreads="12"
>>> maxThreads="18" acceptCount="282"
>>> enableLookups="false" redirectPort="8443"
>>> connectionTimeout="20000"
>>> disableUploadTimeout="true" />
>>>
>>> 2. Use an Apache 2.2.4 with mod_proxy_ajp with a
>>> configuration like:
>>>
>>> <Proxy balancer://ajpWorker>
>>> BalancerMember ajp://localhost:8010 min=16 max=300 smax=40
>>> ttl=900 keepalive=Off timeout=900
>>> </Proxy>
>>>
>>> RewriteEngine on
>>> RewriteRule ^(/TestApp/(.*\.jsp(.*)|servlet/.*|.*\.jar))$
>>> balancer://ajpWorker$1 [P]
>>>
>>> (on Windows in this case; similar results can be obtained on Linux
>>> at least)
>>>
>>> 3. Use a simple test JSP page (placed in a web app
>>> containing nothing
>>> else):
>>>
>>> <%@page session="false"
>>> %><%@page contentType="text/html" pageEncoding="UTF-8"
>>> %><%!
>>> private static final String titleString = "Sleepy
>>> Test JSP Page";
>>> %><html>
>>> <head>
>>> <%
>>> String sleepSeconds = request.getParameter( "secs" );
>>> if ( sleepSeconds == null )
>>> sleepSeconds = "1";
>>> long secsToSleep = Long.parseLong( sleepSeconds );
>>> Thread.sleep( 1000L * secsToSleep );
>>> %>
>>> <meta http-equiv="Content-Type" content="text/html;
>>> charset=UTF-8"/>
>>> <title><%=titleString%></title>
>>> </head>
>>> <body>
>>> <center>
>>> <h2><%=titleString%>: SUCCESS!</h2>
>>> [Slept <%= secsToSleep %> seconds.]
>>> </center>
>>> </body>
>>> </html>
>>>
>>> 4. Hit the page with ab
>>> * First, test direct Tomcat connections:
>>> o ab -n 24 -c 24 -A wcadmin:wcadmin
>>> http://hostname:*8080*/TestApp/test.jsp?secs=3
>>> + Result: All requests complete sucessfully in
>>> 6118 ms.
>>> * Second, test connections via Apache:
>>> o ab -n 24 -c 24 -A wcadmin:wcadmin
>>> http://hostname/TestApp/test.jsp?secs=3
>>> + Result: Only 17 requests complete before ab
>>> times out.
>>> * Third, test connections via Apache again soon (under the
>>> BalancerMember 'timeout' seconds) after the last test
>>> o ab -n 24 -c 24 -A wcadmin:wcadmin
>>> http://hostname/TestApp/test.jsp?secs=3
>>> + Result: Only 9 requests complete
>>> before ab times
>>> out.
>>>
>>> Something is clearly /horribly/ wrong with the handling of any
>>> concurrency over 'maxThreads' in this case. Even so, there
>>> seems to be
>>> some sort of "off-by-one" error in that only 17 requests
>>> complete, not
>>> 18. Worse, this has a lingering effect that decreases
>>> Tomcat's ability
>>> to concurrent requests thereafter (with this impact seemingly
>>> being much
>>> worse the longer the BalancerMember timeout is set to -- and we have
>>> some very long running requests and thus need this timeout to
>>> be /very/
>>> large).
>>>
>>> This is not the only ill effect we've seen when hitting Tomcat 5.5.24
>>> with concurrent requests in this manner, but it is a good
>>> place to start.
>>>
>>> As for the native connector, it just works here. So why don't I just
>>> use it? Well, first off, we have to support Tomcat on
>>> Windows (32 and
>>> 64-bit), Linux, Solaris, HPUX (PA-RISC and Itanium), and AIX. We've
>>> been unable to get the connector built on some of these
>>> platforms and on
>>> some we can't get the resulting binary to function (specifically on
>>> AIX). Further, we had some stability issues with the native
>>> connector
>>> in the past and had considered the Java connector the safest, if not
>>> fastest, option -- and to a degree given that everything is
>>> Java I still
>>> feel that's the case. Finally, however, this connector should just
>>> plain work. Tomcat shouldn't be a cripple unless/until you manage to
>>> build a native connector for your platform.
>>>
>>> Any troubleshooting and/or debugging ideas (e.g. where
>>> exactly to place
>>> breakpoints, what logs to turn on, etc, etc) would be
>>> /greatly/ appreciated.
>>>
>>> --
>>> Jess Holle
>>> jessh@ptc.com
>>>
>>> P.S. If this should go to the user's mailing list instead
>>> that's fine,
>>> but this really seemed like a developer-oriented issue to me.
>>>
>>>
>>>
>>
>>
>>
>> This message is intended only for the use of the person(s) listed above
>> as the intended recipient(s), and may contain information that is
>> PRIVILEGED and CONFIDENTIAL. If you are not an intended recipient, you
>> may not read, copy, or distribute this message or any attachment. If you
>> received this communication in error, please notify us immediately by
>> e-mail and then delete all copies of this message and any attachments.
>>
>> In addition you should be aware that ordinary (unencrypted) e-mail sent
>> through the Internet is not secure. Do not send confidential or sensitive
>> information, such as social security numbers, account numbers, personal
>> identification numbers and passwords, to us via ordinary (unencrypted)
>> e-mail.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: dev-help@tomcat.apache.org
>>
>>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Serious non-native AJP connector issue
Posted by Jess Holle <je...@ptc.com>.
There's no intent to handle this case? Really?
Tomcat should throw an "IllegalArgumentException" whenever 'acceptCount'
is set to anything other than 0 explaining this if this is the case!
If so, this is very unfortunate. We use "maxThreads" as a throttle to
limit the concurrency at this level under the (silly?) assumption that
"acceptCount" behaves as documented in the documentation. [Yes, one
could argue that a separate throttle should be used behind this layer,
which we might have been inclined to do if the documentation said
"acceptCount does not work".]
Further, I've been able to reproduce such issues using a concurrency of
exactly the "maxThreads". At *best* there is an off-by-one error here,
but I've still seen problems in some testing using a concurrency level 2
or 3 less than "maxThreads" -- which leads me to believe there is
something a bit more nefarious going on here. Another of my colleagues
claims that he's encountered issues in simple ab tests quite a ways
below "maxThreads". I've not managed to reproduce this case myself, but
I could certainly focus on this case if the Tomcat contributors have no
intent to fix "acceptCount" behavior for the Java AJP connector.
Is there a firm intent to handle this case in the _/native/_ AJP
connector? If so, then this distinction should be clearly stated (and
an IllegalArgumentException thrown as per above when the native
connector cannot be found/initialized).
Personally I see utility in having "acceptCount" working in the AJP
case. If one exceeds the mod_proxy_ajp hard maximum the client
immediately gets a 503. There are cases when just queuing a few
requests makes more sense than either returning 503's or letting them
rip -- which is where acceptCount comes in. [Granted the acceptCount
numbers below are excessive, but that's another matter...]
--
Jess Holle
jessh@ptc.com
Bill Barker wrote:
> Yes, since you know in advance how many connections you are going to get to
> the AJP connector, you can configure it so that it has enough threads to
> handle all of those connections. That is why it doesn't attempt to handle
> the case when the concurrency goes above maxThreads
>> -----Original Message-----
>> From: Jess Holle [mailto:jessh@ptc.com]
>> Sent: Thursday, June 14, 2007 2:58 PM
>> To: Tomcat Developers List
>> Cc: Dobson, Simon; Wang, Andy; Fenlason, Josh
>> Subject: Serious non-native AJP connector issue
>>
>> We're facing a /serious /issue with the non-native AJP connector.
>>
>> To put it most simply, some requests seem to "get lost" in Tomcat in
>> various cases involving concurrent requests -- and not
>> egregious numbers
>> of concurrent requests, either.
>>
>> For instance,
>>
>> 1. Use a Tomcat 5.5.23 with a configuration like:
>>
>> <Connector port="8010"
>> minSpareThreads="4" maxSpareThreads="12"
>> maxThreads="18" acceptCount="282"
>> tomcatAuthentication="false"
>> useBodyEncodingForURI="true" URIEncoding="UTF-8"
>> enableLookups="false" redirectPort="8443"
>> protocol="AJP/1.3" />
>>
>> (which are intended solely for making it easier to test
>> concurrency
>> issues that to overrun a realistic 'maxThreads' setting) and as a
>> control, similar thread pool settings on the direct HTTP
>> connector:
>>
>> <Connector port="8080" maxHttpHeaderSize="8192"
>> minSpareThreads="4" maxSpareThreads="12"
>> maxThreads="18" acceptCount="282"
>> enableLookups="false" redirectPort="8443"
>> connectionTimeout="20000"
>> disableUploadTimeout="true" />
>>
>> 2. Use an Apache 2.2.4 with mod_proxy_ajp with a
>> configuration like:
>>
>> <Proxy balancer://ajpWorker>
>> BalancerMember ajp://localhost:8010 min=16 max=300 smax=40
>> ttl=900 keepalive=Off timeout=900
>> </Proxy>
>>
>> RewriteEngine on
>> RewriteRule ^(/TestApp/(.*\.jsp(.*)|servlet/.*|.*\.jar))$
>> balancer://ajpWorker$1 [P]
>>
>> (on Windows in this case; similar results can be obtained on Linux
>> at least)
>>
>> 3. Use a simple test JSP page (placed in a web app
>> containing nothing
>> else):
>>
>> <%@page session="false"
>> %><%@page contentType="text/html" pageEncoding="UTF-8"
>> %><%!
>> private static final String titleString = "Sleepy
>> Test JSP Page";
>> %><html>
>> <head>
>> <%
>> String sleepSeconds = request.getParameter( "secs" );
>> if ( sleepSeconds == null )
>> sleepSeconds = "1";
>> long secsToSleep = Long.parseLong( sleepSeconds );
>> Thread.sleep( 1000L * secsToSleep );
>> %>
>> <meta http-equiv="Content-Type" content="text/html;
>> charset=UTF-8"/>
>> <title><%=titleString%></title>
>> </head>
>> <body>
>> <center>
>> <h2><%=titleString%>: SUCCESS!</h2>
>> [Slept <%= secsToSleep %> seconds.]
>> </center>
>> </body>
>> </html>
>>
>> 4. Hit the page with ab
>> * First, test direct Tomcat connections:
>> o ab -n 24 -c 24 -A wcadmin:wcadmin
>> http://hostname:*8080*/TestApp/test.jsp?secs=3
>> + Result: All requests complete sucessfully in
>> 6118 ms.
>> * Second, test connections via Apache:
>> o ab -n 24 -c 24 -A wcadmin:wcadmin
>> http://hostname/TestApp/test.jsp?secs=3
>> + Result: Only 17 requests complete before ab
>> times out.
>> * Third, test connections via Apache again soon (under the
>> BalancerMember 'timeout' seconds) after the last test
>> o ab -n 24 -c 24 -A wcadmin:wcadmin
>> http://hostname/TestApp/test.jsp?secs=3
>> + Result: Only 9 requests complete
>> before ab times
>> out.
>>
>> Something is clearly /horribly/ wrong with the handling of any
>> concurrency over 'maxThreads' in this case. Even so, there
>> seems to be
>> some sort of "off-by-one" error in that only 17 requests
>> complete, not
>> 18. Worse, this has a lingering effect that decreases
>> Tomcat's ability
>> to concurrent requests thereafter (with this impact seemingly
>> being much
>> worse the longer the BalancerMember timeout is set to -- and we have
>> some very long running requests and thus need this timeout to
>> be /very/
>> large).
>>
>> This is not the only ill effect we've seen when hitting Tomcat 5.5.24
>> with concurrent requests in this manner, but it is a good
>> place to start.
>>
>> As for the native connector, it just works here. So why don't I just
>> use it? Well, first off, we have to support Tomcat on
>> Windows (32 and
>> 64-bit), Linux, Solaris, HPUX (PA-RISC and Itanium), and AIX. We've
>> been unable to get the connector built on some of these
>> platforms and on
>> some we can't get the resulting binary to function (specifically on
>> AIX). Further, we had some stability issues with the native
>> connector
>> in the past and had considered the Java connector the safest, if not
>> fastest, option -- and to a degree given that everything is
>> Java I still
>> feel that's the case. Finally, however, this connector should just
>> plain work. Tomcat shouldn't be a cripple unless/until you manage to
>> build a native connector for your platform.
>>
>> Any troubleshooting and/or debugging ideas (e.g. where
>> exactly to place
>> breakpoints, what logs to turn on, etc, etc) would be
>> /greatly/ appreciated.
>>
>> --
>> Jess Holle
>> jessh@ptc.com
>>
>> P.S. If this should go to the user's mailing list instead
>> that's fine,
>> but this really seemed like a developer-oriented issue to me.
>>
>>
>>
>
>
>
> This message is intended only for the use of the person(s) listed above as the intended recipient(s), and may contain information that is PRIVILEGED and CONFIDENTIAL. If you are not an intended recipient, you may not read, copy, or distribute this message or any attachment. If you received this communication in error, please notify us immediately by e-mail and then delete all copies of this message and any attachments.
>
> In addition you should be aware that ordinary (unencrypted) e-mail sent through the Internet is not secure. Do not send confidential or sensitive information, such as social security numbers, account numbers, personal identification numbers and passwords, to us via ordinary (unencrypted) e-mail.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: dev-help@tomcat.apache.org
>
>
RE: Serious non-native AJP connector issue
Posted by Bill Barker <wb...@wilshire.com>.
Yes, since you know in advance how many connections you are going to get to
the AJP connector, you can configure it so that it has enough threads to
handle all of those connections. That is why it doesn't attempt to handle
the case when the concurrency goes above maxThreads.
> -----Original Message-----
> From: Jess Holle [mailto:jessh@ptc.com]
> Sent: Thursday, June 14, 2007 2:58 PM
> To: Tomcat Developers List
> Cc: Dobson, Simon; Wang, Andy; Fenlason, Josh
> Subject: Serious non-native AJP connector issue
>
> We're facing a /serious /issue with the non-native AJP connector.
>
> To put it most simply, some requests seem to "get lost" in Tomcat in
> various cases involving concurrent requests -- and not
> egregious numbers
> of concurrent requests, either.
>
> For instance,
>
> 1. Use a Tomcat 5.5.23 with a configuration like:
>
> <Connector port="8010"
> minSpareThreads="4" maxSpareThreads="12"
> maxThreads="18" acceptCount="282"
> tomcatAuthentication="false"
> useBodyEncodingForURI="true" URIEncoding="UTF-8"
> enableLookups="false" redirectPort="8443"
> protocol="AJP/1.3" />
>
> (which are intended solely for making it easier to test
> concurrency
> issues that to overrun a realistic 'maxThreads' setting) and as a
> control, similar thread pool settings on the direct HTTP
> connector:
>
> <Connector port="8080" maxHttpHeaderSize="8192"
> minSpareThreads="4" maxSpareThreads="12"
> maxThreads="18" acceptCount="282"
> enableLookups="false" redirectPort="8443"
> connectionTimeout="20000"
> disableUploadTimeout="true" />
>
> 2. Use an Apache 2.2.4 with mod_proxy_ajp with a
> configuration like:
>
> <Proxy balancer://ajpWorker>
> BalancerMember ajp://localhost:8010 min=16 max=300 smax=40
> ttl=900 keepalive=Off timeout=900
> </Proxy>
>
> RewriteEngine on
> RewriteRule ^(/TestApp/(.*\.jsp(.*)|servlet/.*|.*\.jar))$
> balancer://ajpWorker$1 [P]
>
> (on Windows in this case; similar results can be obtained on Linux
> at least)
>
> 3. Use a simple test JSP page (placed in a web app
> containing nothing
> else):
>
> <%@page session="false"
> %><%@page contentType="text/html" pageEncoding="UTF-8"
> %><%!
> private static final String titleString = "Sleepy
> Test JSP Page";
> %><html>
> <head>
> <%
> String sleepSeconds = request.getParameter( "secs" );
> if ( sleepSeconds == null )
> sleepSeconds = "1";
> long secsToSleep = Long.parseLong( sleepSeconds );
> Thread.sleep( 1000L * secsToSleep );
> %>
> <meta http-equiv="Content-Type" content="text/html;
> charset=UTF-8"/>
> <title><%=titleString%></title>
> </head>
> <body>
> <center>
> <h2><%=titleString%>: SUCCESS!</h2>
> [Slept <%= secsToSleep %> seconds.]
> </center>
> </body>
> </html>
>
> 4. Hit the page with ab
> * First, test direct Tomcat connections:
> o ab -n 24 -c 24 -A wcadmin:wcadmin
> http://hostname:*8080*/TestApp/test.jsp?secs=3
> + Result: All requests complete sucessfully in
> 6118 ms.
> * Second, test connections via Apache:
> o ab -n 24 -c 24 -A wcadmin:wcadmin
> http://hostname/TestApp/test.jsp?secs=3
> + Result: Only 17 requests complete before ab
> times out.
> * Third, test connections via Apache again soon (under the
> BalancerMember 'timeout' seconds) after the last test
> o ab -n 24 -c 24 -A wcadmin:wcadmin
> http://hostname/TestApp/test.jsp?secs=3
> + Result: Only 9 requests complete
> before ab times
> out.
>
> Something is clearly /horribly/ wrong with the handling of any
> concurrency over 'maxThreads' in this case. Even so, there
> seems to be
> some sort of "off-by-one" error in that only 17 requests
> complete, not
> 18. Worse, this has a lingering effect that decreases
> Tomcat's ability
> to concurrent requests thereafter (with this impact seemingly
> being much
> worse the longer the BalancerMember timeout is set to -- and we have
> some very long running requests and thus need this timeout to
> be /very/
> large).
>
> This is not the only ill effect we've seen when hitting Tomcat 5.5.24
> with concurrent requests in this manner, but it is a good
> place to start.
>
> As for the native connector, it just works here. So why don't I just
> use it? Well, first off, we have to support Tomcat on
> Windows (32 and
> 64-bit), Linux, Solaris, HPUX (PA-RISC and Itanium), and AIX. We've
> been unable to get the connector built on some of these
> platforms and on
> some we can't get the resulting binary to function (specifically on
> AIX). Further, we had some stability issues with the native
> connector
> in the past and had considered the Java connector the safest, if not
> fastest, option -- and to a degree given that everything is
> Java I still
> feel that's the case. Finally, however, this connector should just
> plain work. Tomcat shouldn't be a cripple unless/until you manage to
> build a native connector for your platform.
>
> Any troubleshooting and/or debugging ideas (e.g. where
> exactly to place
> breakpoints, what logs to turn on, etc, etc) would be
> /greatly/ appreciated.
>
> --
> Jess Holle
> jessh@ptc.com
>
> P.S. If this should go to the user's mailing list instead
> that's fine,
> but this really seemed like a developer-oriented issue to me.
>
>
This message is intended only for the use of the person(s) listed above as the intended recipient(s), and may contain information that is PRIVILEGED and CONFIDENTIAL. If you are not an intended recipient, you may not read, copy, or distribute this message or any attachment. If you received this communication in error, please notify us immediately by e-mail and then delete all copies of this message and any attachments.
In addition you should be aware that ordinary (unencrypted) e-mail sent through the Internet is not secure. Do not send confidential or sensitive information, such as social security numbers, account numbers, personal identification numbers and passwords, to us via ordinary (unencrypted) e-mail.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org