You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomcat.apache.org by Jess Holle <je...@ptc.com> on 2007/06/14 23:58:20 UTC

Serious non-native AJP connector issue

We're facing a /serious /issue with the non-native AJP connector.

To put it most simply, some requests seem to "get lost" in Tomcat in 
various cases involving concurrent requests -- and not egregious numbers 
of concurrent requests, either.

For instance,

   1. Use a Tomcat 5.5.23 with a configuration like:

        <Connector port="8010"
                   minSpareThreads="4" maxSpareThreads="12"
    maxThreads="18" acceptCount="282"
                   tomcatAuthentication="false"
    useBodyEncodingForURI="true" URIEncoding="UTF-8"
                   enableLookups="false" redirectPort="8443"
    protocol="AJP/1.3" />

    (which are intended solely for making it easier to test concurrency
    issues that to overrun a realistic 'maxThreads' setting) and as a
    control, similar thread pool settings on the direct HTTP connector:

        <Connector port="8080" maxHttpHeaderSize="8192"
                   minSpareThreads="4" maxSpareThreads="12"
    maxThreads="18" acceptCount="282"
                   enableLookups="false" redirectPort="8443"
                   connectionTimeout="20000" disableUploadTimeout="true" />

   2. Use an Apache 2.2.4 with mod_proxy_ajp with a configuration like:

        <Proxy balancer://ajpWorker>
        BalancerMember ajp://localhost:8010 min=16 max=300 smax=40
        ttl=900 keepalive=Off timeout=900
        </Proxy>

        RewriteEngine on
        RewriteRule ^(/TestApp/(.*\.jsp(.*)|servlet/.*|.*\.jar))$
        balancer://ajpWorker$1 [P]

    (on Windows in this case; similar results can be obtained on Linux
    at least)

   3. Use a simple test JSP page (placed in a web app containing nothing
      else):

        <%@page session="false"
        %><%@page contentType="text/html" pageEncoding="UTF-8"
        %><%!
          private static final String  titleString = "Sleepy Test JSP Page";
        %><html>
        <head>
        <%
          String  sleepSeconds = request.getParameter( "secs" );
          if ( sleepSeconds == null )
            sleepSeconds = "1";
          long  secsToSleep = Long.parseLong( sleepSeconds );
          Thread.sleep( 1000L * secsToSleep );
        %>
        <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
        <title><%=titleString%></title>
        </head>
        <body>
        <center>
        <h2><%=titleString%>: SUCCESS!</h2>
        [Slept <%= secsToSleep %> seconds.]
        </center>
        </body>
        </html>

   4. Hit the page with ab
          * First, test direct Tomcat connections:
                o ab -n 24 -c 24 -A wcadmin:wcadmin
                  http://hostname:*8080*/TestApp/test.jsp?secs=3
                      + Result: All requests complete sucessfully in
                        6118 ms.
          * Second, test connections via Apache:
                o ab -n 24 -c 24 -A wcadmin:wcadmin
                  http://hostname/TestApp/test.jsp?secs=3
                      + Result: Only 17 requests complete before ab
                        times out.
          * Third, test connections via Apache again soon (under the
            BalancerMember 'timeout' seconds) after the last test
                o ab -n 24 -c 24 -A wcadmin:wcadmin
                  http://hostname/TestApp/test.jsp?secs=3
                      + Result: Only 9 requests complete before ab times
                        out.

Something is clearly /horribly/ wrong with the handling of any 
concurrency over 'maxThreads' in this case.  Even so, there seems to be 
some sort of "off-by-one" error in that only 17 requests complete, not 
18.  Worse, this has a lingering effect that decreases Tomcat's ability 
to concurrent requests thereafter (with this impact seemingly being much 
worse the longer the BalancerMember timeout is set to -- and we have 
some very long running requests and thus need this timeout to be /very/ 
large).

This is not the only ill effect we've seen when hitting Tomcat 5.5.24 
with concurrent requests in this manner, but it is a good place to start.

As for the native connector, it just works here.  So why don't I just 
use it?  Well, first off, we have to support Tomcat on Windows (32 and 
64-bit), Linux, Solaris, HPUX (PA-RISC and Itanium), and AIX.  We've 
been unable to get the connector built on some of these platforms and on 
some we can't get the resulting binary to function (specifically on 
AIX).  Further, we had some stability issues with the native connector 
in the past and had considered the Java connector the safest, if not 
fastest, option -- and to a degree given that everything is Java I still 
feel that's the case.  Finally, however, this connector should just 
plain work.  Tomcat shouldn't be a cripple unless/until you manage to 
build a native connector for your platform.

Any troubleshooting and/or debugging ideas (e.g. where exactly to place 
breakpoints, what logs to turn on, etc, etc) would be /greatly/ appreciated.

--
Jess Holle
jessh@ptc.com

P.S. If this should go to the user's mailing list instead that's fine, 
but this really seemed like a developer-oriented issue to me.


Re: Serious non-native AJP connector issue

Posted by Jess Holle <je...@ptc.com>.
Rainer Jung wrote:
> Jess,
>
> I didn't really carefully think about the case of a saturated pool. 
> But nevertheless some hints:
>
> 1) There is always one thread waiting for the accept, which is a usual 
> pool thread. So an offset of one between threeads processing requests 
> and the pool size is normal.
Got that, but I've accounted for that via maxThreads of 51 and 
BalanceMember max of 50.

I'm left wondering why there's an off-by-one beyond this (and thus an 
off-by-2 overall).
> 2) There is no orderly shutdown in AJP13 for a connection, neither 
> from the httpd side, not from the Tomcat side. mod_jk will realize 
> closed connections and reopen them, but I have the impression, that 
> mod_proxy fails a backend when it gets a connection error.
>
> Tomcat in my experience reliably closes a connection and frees the 
> thread, when the web server does.
>
> Most of the time it makes sense to have a connectionTimeout 
> (Milliseconds) and a connection_pool_timeout (seconds, mod_jk) resp. 
> ttl (seconds, mod_proxy) which are in sync. But: they will never 
> harmonize completely, because mod_jk only checks for the need to throw 
> away closed connections during maintenance (every 60 seconds or 
> whatever is configured with worker.maintain) and I think mod_proxy 
> checks the ttl whenever a connection is put back into the pool.
I don't think any of those should be involved in this short test.
> 3) Does it happen with mod_jk too?
I believe so.  I did some mod_jk testing to verify the overall nature of 
the issue remained the same, but I didn't go through all the same 
detailed tests.  I could redo this particular test if that's helpful.
> 4) Weird guesses:
>
> - max is limited with mod_proxy to the number of threads per process 
> configured in your MPM (worker?).
>
> This is 25 by default. So if we want to have an easy to understand 
> scenario, set your threads per process to say 60 and min=max=50 as 
> well as removing the smax and the ttl.. That way 50 connections should 
> be started on startup (per httpd process; check with netstat), and we 
> shouldn't see any resizing during your ab test. Now start your ab test 
> before Tomcat will timeout (I remember that connectionTimeout had some 
> default value, even if you don't set it, but it is in the minutes).
My Apache MaxClients is 300 and this is on Windows so I only have 1 
worker process.
> If you don't run into trouble then, we know your observation has to do 
> with resizing of connection pools.
>
> *Maybe*: ab is too fast and can come back with new requests faster, 
> than the connections go back into the pool after the end of a request 
> in httpd. Not very reasonable, but possible. Of cource the pool is 
> synchronized and the lock doesn't need to behave fair, i.e. when it 
> gets contended, it's not clear if the oldest waiting thread gets it 
> first.
I believe I disproved this at some point by running 2 tests with -n 50 
and -c 50 manually back to back, but I'd have to re-test to be sure.  
[I'm wishing I'd taken better notes of various results...]

Apart from this weird edge condition (an off-by-2 isn't that devastating 
if it stays "2" in all cases), the thing that gets me overall is that 
the documentation really makes it sound like "acceptCount" works like a 
fair queue and that there's no harm in exceeding maxThreads except that 
requests will be queued.  As Bill suggested, I should come up with 
suggested patches to the documentation -- I'm just not yet confident 
enough in my understanding to propose such patches.  At this point all I 
can propose is strong warning verbage around maxThreads and acceptCount 
regarding their behavior for the Java AJP connector.

--
Jess Holle

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org


Re: Serious non-native AJP connector issue

Posted by Rainer Jung <ra...@kippdata.de>.
Jess,

I didn't really carefully think about the case of a saturated pool. But 
nevertheless some hints:

1) There is always one thread waiting for the accept, which is a usual 
pool thread. So an offset of one between threeads processing requests 
and the pool size is normal.

2) There is no orderly shutdown in AJP13 for a connection, neither from 
the httpd side, not from the Tomcat side. mod_jk will realize closed 
connections and reopen them, but I have the impression, that mod_proxy 
fails a backend when it gets a connection error.

Tomcat in my experience reliably closes a connection and frees the 
thread, when the web server does.

Most of the time it makes sense to have a connectionTimeout 
(Milliseconds) and a connection_pool_timeout (seconds, mod_jk) resp. ttl 
(seconds, mod_proxy) which are in sync. But: they will never harmonize 
completely, because mod_jk only checks for the need to throw away closed 
connections during maintenance (every 60 seconds or whatever is 
configured with worker.maintain) and I think mod_proxy checks the ttl 
whenever a connection is put back into the pool.

3) Does it happen with mod_jk too?

4) Weird guesses:

- max is limited with mod_proxy to the number of threads per process 
configured in your MPM (worker?).

This is 25 by default. So if we want to have an easy to understand 
scenario, set your threads per process to say 60 and min=max=50 as well 
as removing the smax and the ttl.. That way 50 connections should be 
started on startup (per httpd process; check with netstat), and we 
shouldn't see any resizing during your ab test. Now start your ab test 
before Tomcat will timeout (I remember that connectionTimeout had some 
default value, even if you don't set it, but it is in the minutes).

If you don't run into trouble then, we know your observation has to do 
with resizing of connection pools.

*Maybe*: ab is too fast and can come back with new requests faster, than 
the connections go back into the pool after the end of a request in 
httpd. Not very reasonable, but possible. Of cource the pool is 
synchronized and the lock doesn't need to behave fair, i.e. when it gets 
contended, it's not clear if the oldest waiting thread gets it first.

Regards,

Rainer

Jess Holle wrote:
> Hmmmm....
> 
> I just redid my tests with:
> 
>    BalancerMember ajp://localhost:8010 min=15 max=*50* smax=30 ttl=900
>    keepalive=Off timeout=900
> 
> and
> 
>        <Connector port="8010"
>                   minSpareThreads="15" maxSpareThreads="30"
>    maxThreads="*51*" acceptCount="*0*"
>                   tomcatAuthentication="false"
>    useBodyEncodingForURI="true" URIEncoding="UTF-8"
>                   enableLookups="false" redirectPort="8443"
>    protocol="AJP/1.3" />
> 
> and
> 
>    ab -n 100 -c *50* http://jessh03l.ptcnet.ptc.com/TestApp/test.jsp?secs=3
> 
> I (after about 3 seconds) get
> 
>    SEVERE: All threads (51) are currently busy, waiting. Increase
>    maxThreads (51) or check the servlet status
> 
> and I eventually get exactly 1 lost request.  I'm baffled as to why this 
> can occur.
> 
> Something's still doesn't seem quite right here.
> 
> What's even weirder is that I only get this issue with the first ab run 
> after restarting Tomcat.  If I do the same test again (any number of 
> times) I don't lose any requests.
> 
> I can get the same result by restarting and doing 2 ab runs with "-n 
> 100" in fairly short secession, so this isn't some ab error.  By "fairly 
> short", I don't mean very short -- I left a good 5 seconds between runs.
> 
> I find that using a max of 49 in Apache seems to work reliably, but I'm 
> struggling to understand: (1) why I have to have 2 more Tomcat connector 
> threads allowed than the number of connections I can possibly have and 
> (2) whether "2" is always the safe buffer or whether I'll need 9 on an 8 
> CPU box or some such.
> 
> -- 
> Jess Holle

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org


Re: Serious non-native AJP connector issue

Posted by Jess Holle <je...@ptc.com>.
Hmmmm....

I just redid my tests with:

    BalancerMember ajp://localhost:8010 min=15 max=*50* smax=30 ttl=900
    keepalive=Off timeout=900

and

        <Connector port="8010"
                   minSpareThreads="15" maxSpareThreads="30"
    maxThreads="*51*" acceptCount="*0*"
                   tomcatAuthentication="false"
    useBodyEncodingForURI="true" URIEncoding="UTF-8"
                   enableLookups="false" redirectPort="8443"
    protocol="AJP/1.3" />

and

    ab -n 100 -c *50* http://jessh03l.ptcnet.ptc.com/TestApp/test.jsp?secs=3

I (after about 3 seconds) get

    SEVERE: All threads (51) are currently busy, waiting. Increase
    maxThreads (51) or check the servlet status

and I eventually get exactly 1 lost request.  I'm baffled as to why this 
can occur.

Something's still doesn't seem quite right here.

What's even weirder is that I only get this issue with the first ab run 
after restarting Tomcat.  If I do the same test again (any number of 
times) I don't lose any requests.

I can get the same result by restarting and doing 2 ab runs with "-n 
100" in fairly short secession, so this isn't some ab error.  By "fairly 
short", I don't mean very short -- I left a good 5 seconds between runs.

I find that using a max of 49 in Apache seems to work reliably, but I'm 
struggling to understand: (1) why I have to have 2 more Tomcat connector 
threads allowed than the number of connections I can possibly have and 
(2) whether "2" is always the safe buffer or whether I'll need 9 on an 8 
CPU box or some such.

--
Jess Holle


Re: Serious non-native AJP connector issue

Posted by Jess Holle <je...@ptc.com>.
Bill Barker wrote:
> It seems that we lost building the NIO/AJP connector in Tomcat, but since i 
> recognize you from dev@httpd, you will have seen that already :).  Last time 
> I tested, the experimental NIO/AJP connector performed better than either 
> the default or the APR connector on Solaris (surprise, surprise), and came 
> in dead last on Windows.
>   
Thanks for all the helpful information, Bill.

I wonder if the results of for the NIO/AJP connector would be any 
different on Windows with a later JVM (e.g. Java 6 Update 1).  I would 
*hope* that Sun would fix the NIO issues on Windows to the point that it 
would out-perform Java-Blocking-IO/AJP.

--
Jess Holle

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org


Re: Serious non-native AJP connector issue

Posted by Bill Barker <wb...@wilshire.com>.
"Jess Holle" <je...@ptc.com> wrote in message news:4671FC01.60408@ptc.com...
> Okay that's all starting to make some sense, but it hard to see how
> someone would come to this understanding from the documentation.
>
> I also don't see how "|connectionTimeout" will help here in that the
> documentation says:
> |
>
>    The number of milliseconds this *Connector* will wait, after
>    accepting a connection, for the request URI line to be presented.
>    The default value is infinite (i.e. no timeout).
>

In practice, this is a timeout on a wait on a keep-alive connection (since 
request-body processing usually happens fast).  If this particular 
connection in the pool isn't reused in the time allowed, then Tomcat will 
close it and put the thread back in the queue to process new connections. 
Using CPing/CPong is recommended in this case to allow httpd to understand 
that Tomcat has hung up the phone.  So, if after the first request, httpd 
chooses to not reuse this connection for a long time, then the thread can 
recycle itself.

> If a connection is formed between Apache and Tomcat for 24 requests and
> 17 are immediately processed due to a maxThreads limit of 18, how would
> setting a low "connectionTimeout" help the 17 threads process the other
> 7 requests?
>

It would cause the TC threads listening for a new request to believe that it 
will never come, and they will throw themselves back into the pool.

> I'm trying to understand if there is any useful throttling configuration
> in which the Java AJP connector has a maxThreads less than Apache's max
> AJP connections -- plus 1.  Put another way, I'm not quite understanding
> any useful "acceptCount" scenario from the "connectionTimeout"
> description above.
>

Using acceptCount is normally useless.  The case where it is is when you 
suddenly get a very very large number of hits to your app faster than Tomcat 
can handle.

> I know the Java AJP connector uses a thread-per-connection model.  I had
> assumed that the "maxThreads" meant maximum /active request processing
> /threads (e.g. like some old Tomcat releases used to call this
> "maxProcessors") and that connections covered by "acceptCount" were
> still allocated threads and were still accepted (as the parameter name
> implies).  I'd further assumed that a fair blocking queue arrangement
> would allow "maxThreads" connection threads to run and keep the others
> waiting until their turn.  I don't see anything in the documentation to
> the contrary of this.  Further I'm trying to understand any sort of
> arrangement with "connectionTimeout" that would give this level of
> utility to "acceptCount".  As it stands it seems like users can easily
> produce devastatingly bad behavior by making assumptions about
> "acceptCount" that seem quite logical from the documentation.
>

The acceptCount is only useful to handle unexpected floods of requests. 
Using the connectionTimeout means that the Tomcat thread won't just sit and 
listen forever for that particler socket connection to send another request, 
and will be freed up to be re-assigned to another socket connection.  With 
any reasonable setting, the connectionTimeout is just how long Tomcat will 
maintain a keep-alive with httpd (which is very different from how long 
httpd will keep a keep-alive with the client).

> Obviously the APR connector can do much better by /not/ allocating
> threads to connections beyond "maxThreads".  That's great, but juggling
> native builds for many different OS's can be a real issue (which is why
> I still hold out hope that the NIO connection will come through with
> something better than the non-NIO Java connector's performance even if
> it is not APR-level performance).
>

It seems that we lost building the NIO/AJP connector in Tomcat, but since i 
recognize you from dev@httpd, you will have seen that already :).  Last time 
I tested, the experimental NIO/AJP connector performed better than either 
the default or the APR connector on Solaris (surprise, surprise), and came 
in dead last on Windows.

> I don't mean to be argumentative -- I'm just really struggling to
> understand and hopefully preventing future misunderstandings through
> more clarity in the documentation.
>

With the documentation, as with the code, patches are always welcome :).


> --
> Jess Holle
>
> Bill Barker wrote:
>> "Jess Holle" <je...@ptc.com> wrote in message
>> news:4671E278.8020204@ptc.com...
>>
>>> There's no intent to handle this case?  Really?
>>>
>>> Tomcat should throw an "IllegalArgumentException" whenever 'acceptCount'
>>> is set to anything other than 0 explaining this if this is the case!
>>>
>>> If so, this is very unfortunate.  We use "maxThreads" as a throttle to
>>> limit the concurrency at this level under the (silly?) assumption that
>>> "acceptCount" behaves as documented in the documentation.  [Yes, one
>>> could argue that a separate throttle should be used behind this layer,
>>> which we might have been inclined to do if the documentation said
>>> "acceptCount does not work".]
>>>
>>>
>>
>> You are misunderstanding how AJP works.  Since you don't have a
>> connectionTimeout on the <Connector />, the connections to httpd stay 
>> alive
>> waiting to get another request on the same socket.  As a result, there 
>> won't
>> be any free Threads to handle a new connection so it doesn't matter what
>> acceptCount is.  Their isn't anyone there to accept them.
>>
>
> 




---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org


Re: Serious non-native AJP connector issue

Posted by Jess Holle <je...@ptc.com>.
Okay that's all starting to make some sense, but it hard to see how 
someone would come to this understanding from the documentation.

I also don't see how "|connectionTimeout" will help here in that the 
documentation says:
|

    The number of milliseconds this *Connector* will wait, after
    accepting a connection, for the request URI line to be presented.
    The default value is infinite (i.e. no timeout).

If a connection is formed between Apache and Tomcat for 24 requests and 
17 are immediately processed due to a maxThreads limit of 18, how would 
setting a low "connectionTimeout" help the 17 threads process the other 
7 requests?

I'm trying to understand if there is any useful throttling configuration 
in which the Java AJP connector has a maxThreads less than Apache's max 
AJP connections -- plus 1.  Put another way, I'm not quite understanding 
any useful "acceptCount" scenario from the "connectionTimeout" 
description above.

I know the Java AJP connector uses a thread-per-connection model.  I had 
assumed that the "maxThreads" meant maximum /active request processing 
/threads (e.g. like some old Tomcat releases used to call this 
"maxProcessors") and that connections covered by "acceptCount" were 
still allocated threads and were still accepted (as the parameter name 
implies).  I'd further assumed that a fair blocking queue arrangement 
would allow "maxThreads" connection threads to run and keep the others 
waiting until their turn.  I don't see anything in the documentation to 
the contrary of this.  Further I'm trying to understand any sort of 
arrangement with "connectionTimeout" that would give this level of 
utility to "acceptCount".  As it stands it seems like users can easily 
produce devastatingly bad behavior by making assumptions about 
"acceptCount" that seem quite logical from the documentation.

Obviously the APR connector can do much better by /not/ allocating 
threads to connections beyond "maxThreads".  That's great, but juggling 
native builds for many different OS's can be a real issue (which is why 
I still hold out hope that the NIO connection will come through with 
something better than the non-NIO Java connector's performance even if 
it is not APR-level performance).

I don't mean to be argumentative -- I'm just really struggling to 
understand and hopefully preventing future misunderstandings through 
more clarity in the documentation.

--
Jess Holle

Bill Barker wrote:
> "Jess Holle" <je...@ptc.com> wrote in message 
> news:4671E278.8020204@ptc.com...
>   
>> There's no intent to handle this case?  Really?
>>
>> Tomcat should throw an "IllegalArgumentException" whenever 'acceptCount'
>> is set to anything other than 0 explaining this if this is the case!
>>
>> If so, this is very unfortunate.  We use "maxThreads" as a throttle to
>> limit the concurrency at this level under the (silly?) assumption that
>> "acceptCount" behaves as documented in the documentation.  [Yes, one
>> could argue that a separate throttle should be used behind this layer,
>> which we might have been inclined to do if the documentation said
>> "acceptCount does not work".]
>>
>>     
>
> You are misunderstanding how AJP works.  Since you don't have a 
> connectionTimeout on the <Connector />, the connections to httpd stay alive 
> waiting to get another request on the same socket.  As a result, there won't 
> be any free Threads to handle a new connection so it doesn't matter what 
> acceptCount is.  Their isn't anyone there to accept them.
>   


Re: Serious non-native AJP connector issue

Posted by Bill Barker <wb...@wilshire.com>.
"Jess Holle" <je...@ptc.com> wrote in message 
news:4671E278.8020204@ptc.com...
> There's no intent to handle this case?  Really?
>
> Tomcat should throw an "IllegalArgumentException" whenever 'acceptCount'
> is set to anything other than 0 explaining this if this is the case!
>
> If so, this is very unfortunate.  We use "maxThreads" as a throttle to
> limit the concurrency at this level under the (silly?) assumption that
> "acceptCount" behaves as documented in the documentation.  [Yes, one
> could argue that a separate throttle should be used behind this layer,
> which we might have been inclined to do if the documentation said
> "acceptCount does not work".]
>

You are misunderstanding how AJP works.  Since you don't have a 
connectionTimeout on the <Connector />, the connections to httpd stay alive 
waiting to get another request on the same socket.  As a result, there won't 
be any free Threads to handle a new connection so it doesn't matter what 
acceptCount is.  Their isn't anyone there to accept them.

> Further, I've been able to reproduce such issues using a concurrency of
> exactly the "maxThreads".  At *best* there is an off-by-one error here,
> but I've still seen problems in some testing using a concurrency level 2
> or 3 less than "maxThreads" -- which leads me to believe there is
> something a bit more nefarious going on here.  Another of my colleagues
> claims that he's encountered issues in simple ab tests quite a ways
> below "maxThreads".  I've not managed to reproduce this case myself, but
> I could certainly focus on this case if the Tomcat contributors have no
> intent to fix "acceptCount" behavior for the Java AJP connector.
>

The thread waiting to accept new connections counts as a thread for the 
purposes of maxThreads, so yes it is one less.  And, I can save you the 
trouble of investigating, since as long as your maxThreads <= pool size in 
httpd you will see this sort of thing.

> Is there a firm intent to handle this case in the _/native/_ AJP
> connector?  If so, then this distinction should be clearly stated (and
> an IllegalArgumentException thrown as per above when the native
> connector cannot be found/initialized).
>

The APR/AJP connector (as well as the experimental NIO/AJP connector) use a 
many-to-one connection-to-thread setup so you can have a large number of 
connections sitting and doing nothing and use relatively few threads.  With 
the default Java connector the mapping is one-to-one.

> Personally I see utility in having "acceptCount" working in the AJP
> case.  If one exceeds the mod_proxy_ajp hard maximum the client
> immediately gets a 503.  There are cases when just queuing a few
> requests makes more sense than either returning 503's or letting them
> rip -- which is where acceptCount comes in.  [Granted the acceptCount
> numbers below are excessive, but that's another matter...]
>

It's not that acceptCount doesn't work, it's that without a 
connectionTimeout each thread waits forever listening on it's connection for 
the next request to come in.  With a connectionTimeout, once httpd hasn't 
sent another request on this connection after the timeout, the thread is 
recycled and can handle a new connection.

> --
> Jess Holle
> jessh@ptc.com
>
> Bill Barker wrote:
>> Yes, since you know in advance how many connections you are going to get 
>> to
>> the AJP connector, you can configure it so that it has enough threads to
>> handle all of those connections.  That is why it doesn't attempt to 
>> handle
>> the case when the concurrency goes above maxThreads
>>> -----Original Message-----
>>> From: Jess Holle [mailto:jessh@ptc.com]
>>> Sent: Thursday, June 14, 2007 2:58 PM
>>> To: Tomcat Developers List
>>> Cc: Dobson, Simon; Wang, Andy; Fenlason, Josh
>>> Subject: Serious non-native AJP connector issue
>>>
>>> We're facing a /serious /issue with the non-native AJP connector.
>>>
>>> To put it most simply, some requests seem to "get lost" in Tomcat in
>>> various cases involving concurrent requests -- and not
>>> egregious numbers
>>> of concurrent requests, either.
>>>
>>> For instance,
>>>
>>>    1. Use a Tomcat 5.5.23 with a configuration like:
>>>
>>>         <Connector port="8010"
>>>                    minSpareThreads="4" maxSpareThreads="12"
>>>     maxThreads="18" acceptCount="282"
>>>                    tomcatAuthentication="false"
>>>     useBodyEncodingForURI="true" URIEncoding="UTF-8"
>>>                    enableLookups="false" redirectPort="8443"
>>>     protocol="AJP/1.3" />
>>>
>>>     (which are intended solely for making it easier to test
>>> concurrency
>>>     issues that to overrun a realistic 'maxThreads' setting) and as a
>>>     control, similar thread pool settings on the direct HTTP
>>> connector:
>>>
>>>         <Connector port="8080" maxHttpHeaderSize="8192"
>>>                    minSpareThreads="4" maxSpareThreads="12"
>>>     maxThreads="18" acceptCount="282"
>>>                    enableLookups="false" redirectPort="8443"
>>>                    connectionTimeout="20000"
>>> disableUploadTimeout="true" />
>>>
>>>    2. Use an Apache 2.2.4 with mod_proxy_ajp with a
>>> configuration like:
>>>
>>>         <Proxy balancer://ajpWorker>
>>>         BalancerMember ajp://localhost:8010 min=16 max=300 smax=40
>>>         ttl=900 keepalive=Off timeout=900
>>>         </Proxy>
>>>
>>>         RewriteEngine on
>>>         RewriteRule ^(/TestApp/(.*\.jsp(.*)|servlet/.*|.*\.jar))$
>>>         balancer://ajpWorker$1 [P]
>>>
>>>     (on Windows in this case; similar results can be obtained on Linux
>>>     at least)
>>>
>>>    3. Use a simple test JSP page (placed in a web app
>>> containing nothing
>>>       else):
>>>
>>>         <%@page session="false"
>>>         %><%@page contentType="text/html" pageEncoding="UTF-8"
>>>         %><%!
>>>           private static final String  titleString = "Sleepy
>>> Test JSP Page";
>>>         %><html>
>>>         <head>
>>>         <%
>>>           String  sleepSeconds = request.getParameter( "secs" );
>>>           if ( sleepSeconds == null )
>>>             sleepSeconds = "1";
>>>           long  secsToSleep = Long.parseLong( sleepSeconds );
>>>           Thread.sleep( 1000L * secsToSleep );
>>>         %>
>>>         <meta http-equiv="Content-Type" content="text/html;
>>> charset=UTF-8"/>
>>>         <title><%=titleString%></title>
>>>         </head>
>>>         <body>
>>>         <center>
>>>         <h2><%=titleString%>: SUCCESS!</h2>
>>>         [Slept <%= secsToSleep %> seconds.]
>>>         </center>
>>>         </body>
>>>         </html>
>>>
>>>    4. Hit the page with ab
>>>           * First, test direct Tomcat connections:
>>>                 o ab -n 24 -c 24 -A wcadmin:wcadmin
>>>                   http://hostname:*8080*/TestApp/test.jsp?secs=3
>>>                       + Result: All requests complete sucessfully in
>>>                         6118 ms.
>>>           * Second, test connections via Apache:
>>>                 o ab -n 24 -c 24 -A wcadmin:wcadmin
>>>                   http://hostname/TestApp/test.jsp?secs=3
>>>                       + Result: Only 17 requests complete before ab
>>>                         times out.
>>>           * Third, test connections via Apache again soon (under the
>>>             BalancerMember 'timeout' seconds) after the last test
>>>                 o ab -n 24 -c 24 -A wcadmin:wcadmin
>>>                   http://hostname/TestApp/test.jsp?secs=3
>>>                       + Result: Only 9 requests complete
>>> before ab times
>>>                         out.
>>>
>>> Something is clearly /horribly/ wrong with the handling of any
>>> concurrency over 'maxThreads' in this case.  Even so, there
>>> seems to be
>>> some sort of "off-by-one" error in that only 17 requests
>>> complete, not
>>> 18.  Worse, this has a lingering effect that decreases
>>> Tomcat's ability
>>> to concurrent requests thereafter (with this impact seemingly
>>> being much
>>> worse the longer the BalancerMember timeout is set to -- and we have
>>> some very long running requests and thus need this timeout to
>>> be /very/
>>> large).
>>>
>>> This is not the only ill effect we've seen when hitting Tomcat 5.5.24
>>> with concurrent requests in this manner, but it is a good
>>> place to start.
>>>
>>> As for the native connector, it just works here.  So why don't I just
>>> use it?  Well, first off, we have to support Tomcat on
>>> Windows (32 and
>>> 64-bit), Linux, Solaris, HPUX (PA-RISC and Itanium), and AIX.  We've
>>> been unable to get the connector built on some of these
>>> platforms and on
>>> some we can't get the resulting binary to function (specifically on
>>> AIX).  Further, we had some stability issues with the native
>>> connector
>>> in the past and had considered the Java connector the safest, if not
>>> fastest, option -- and to a degree given that everything is
>>> Java I still
>>> feel that's the case.  Finally, however, this connector should just
>>> plain work.  Tomcat shouldn't be a cripple unless/until you manage to
>>> build a native connector for your platform.
>>>
>>> Any troubleshooting and/or debugging ideas (e.g. where
>>> exactly to place
>>> breakpoints, what logs to turn on, etc, etc) would be
>>> /greatly/ appreciated.
>>>
>>> --
>>> Jess Holle
>>> jessh@ptc.com
>>>
>>> P.S. If this should go to the user's mailing list instead
>>> that's fine,
>>> but this really seemed like a developer-oriented issue to me.
>>>
>>>
>>>
>>
>>
>>
>> This message is intended only for the use of the person(s) listed above 
>> as the intended recipient(s), and may contain information that is 
>> PRIVILEGED and CONFIDENTIAL.  If you are not an intended recipient, you 
>> may not read, copy, or distribute this message or any attachment. If you 
>> received this communication in error, please notify us immediately by 
>> e-mail and then delete all copies of this message and any attachments.
>>
>> In addition you should be aware that ordinary (unencrypted) e-mail sent 
>> through the Internet is not secure. Do not send confidential or sensitive 
>> information, such as social security numbers, account numbers, personal 
>> identification numbers and passwords, to us via ordinary (unencrypted) 
>> e-mail.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: dev-help@tomcat.apache.org
>>
>>
>
> 




---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org


Re: Serious non-native AJP connector issue

Posted by Jess Holle <je...@ptc.com>.
There's no intent to handle this case?  Really?

Tomcat should throw an "IllegalArgumentException" whenever 'acceptCount' 
is set to anything other than 0 explaining this if this is the case!

If so, this is very unfortunate.  We use "maxThreads" as a throttle to 
limit the concurrency at this level under the (silly?) assumption that 
"acceptCount" behaves as documented in the documentation.  [Yes, one 
could argue that a separate throttle should be used behind this layer, 
which we might have been inclined to do if the documentation said 
"acceptCount does not work".]

Further, I've been able to reproduce such issues using a concurrency of 
exactly the "maxThreads".  At *best* there is an off-by-one error here, 
but I've still seen problems in some testing using a concurrency level 2 
or 3 less than "maxThreads" -- which leads me to believe there is 
something a bit more nefarious going on here.  Another of my colleagues 
claims that he's encountered issues in simple ab tests quite a ways 
below "maxThreads".  I've not managed to reproduce this case myself, but 
I could certainly focus on this case if the Tomcat contributors have no 
intent to fix "acceptCount" behavior for the Java AJP connector.

Is there a firm intent to handle this case in the _/native/_ AJP 
connector?  If so, then this distinction should be clearly stated (and 
an IllegalArgumentException thrown as per above when the native 
connector cannot be found/initialized).

Personally I see utility in having "acceptCount" working in the AJP 
case.  If one exceeds the mod_proxy_ajp hard maximum the client 
immediately gets a 503.  There are cases when just queuing a few 
requests makes more sense than either returning 503's or letting them 
rip -- which is where acceptCount comes in.  [Granted the acceptCount 
numbers below are excessive, but that's another matter...]

--
Jess Holle
jessh@ptc.com

Bill Barker wrote:
> Yes, since you know in advance how many connections you are going to get to
> the AJP connector, you can configure it so that it has enough threads to
> handle all of those connections.  That is why it doesn't attempt to handle
> the case when the concurrency goes above maxThreads
>> -----Original Message-----
>> From: Jess Holle [mailto:jessh@ptc.com] 
>> Sent: Thursday, June 14, 2007 2:58 PM
>> To: Tomcat Developers List
>> Cc: Dobson, Simon; Wang, Andy; Fenlason, Josh
>> Subject: Serious non-native AJP connector issue
>>
>> We're facing a /serious /issue with the non-native AJP connector.
>>
>> To put it most simply, some requests seem to "get lost" in Tomcat in 
>> various cases involving concurrent requests -- and not 
>> egregious numbers 
>> of concurrent requests, either.
>>
>> For instance,
>>
>>    1. Use a Tomcat 5.5.23 with a configuration like:
>>
>>         <Connector port="8010"
>>                    minSpareThreads="4" maxSpareThreads="12"
>>     maxThreads="18" acceptCount="282"
>>                    tomcatAuthentication="false"
>>     useBodyEncodingForURI="true" URIEncoding="UTF-8"
>>                    enableLookups="false" redirectPort="8443"
>>     protocol="AJP/1.3" />
>>
>>     (which are intended solely for making it easier to test 
>> concurrency
>>     issues that to overrun a realistic 'maxThreads' setting) and as a
>>     control, similar thread pool settings on the direct HTTP 
>> connector:
>>
>>         <Connector port="8080" maxHttpHeaderSize="8192"
>>                    minSpareThreads="4" maxSpareThreads="12"
>>     maxThreads="18" acceptCount="282"
>>                    enableLookups="false" redirectPort="8443"
>>                    connectionTimeout="20000" 
>> disableUploadTimeout="true" />
>>
>>    2. Use an Apache 2.2.4 with mod_proxy_ajp with a 
>> configuration like:
>>
>>         <Proxy balancer://ajpWorker>
>>         BalancerMember ajp://localhost:8010 min=16 max=300 smax=40
>>         ttl=900 keepalive=Off timeout=900
>>         </Proxy>
>>
>>         RewriteEngine on
>>         RewriteRule ^(/TestApp/(.*\.jsp(.*)|servlet/.*|.*\.jar))$
>>         balancer://ajpWorker$1 [P]
>>
>>     (on Windows in this case; similar results can be obtained on Linux
>>     at least)
>>
>>    3. Use a simple test JSP page (placed in a web app 
>> containing nothing
>>       else):
>>
>>         <%@page session="false"
>>         %><%@page contentType="text/html" pageEncoding="UTF-8"
>>         %><%!
>>           private static final String  titleString = "Sleepy 
>> Test JSP Page";
>>         %><html>
>>         <head>
>>         <%
>>           String  sleepSeconds = request.getParameter( "secs" );
>>           if ( sleepSeconds == null )
>>             sleepSeconds = "1";
>>           long  secsToSleep = Long.parseLong( sleepSeconds );
>>           Thread.sleep( 1000L * secsToSleep );
>>         %>
>>         <meta http-equiv="Content-Type" content="text/html; 
>> charset=UTF-8"/>
>>         <title><%=titleString%></title>
>>         </head>
>>         <body>
>>         <center>
>>         <h2><%=titleString%>: SUCCESS!</h2>
>>         [Slept <%= secsToSleep %> seconds.]
>>         </center>
>>         </body>
>>         </html>
>>
>>    4. Hit the page with ab
>>           * First, test direct Tomcat connections:
>>                 o ab -n 24 -c 24 -A wcadmin:wcadmin
>>                   http://hostname:*8080*/TestApp/test.jsp?secs=3
>>                       + Result: All requests complete sucessfully in
>>                         6118 ms.
>>           * Second, test connections via Apache:
>>                 o ab -n 24 -c 24 -A wcadmin:wcadmin
>>                   http://hostname/TestApp/test.jsp?secs=3
>>                       + Result: Only 17 requests complete before ab
>>                         times out.
>>           * Third, test connections via Apache again soon (under the
>>             BalancerMember 'timeout' seconds) after the last test
>>                 o ab -n 24 -c 24 -A wcadmin:wcadmin
>>                   http://hostname/TestApp/test.jsp?secs=3
>>                       + Result: Only 9 requests complete 
>> before ab times
>>                         out.
>>
>> Something is clearly /horribly/ wrong with the handling of any 
>> concurrency over 'maxThreads' in this case.  Even so, there 
>> seems to be 
>> some sort of "off-by-one" error in that only 17 requests 
>> complete, not 
>> 18.  Worse, this has a lingering effect that decreases 
>> Tomcat's ability 
>> to concurrent requests thereafter (with this impact seemingly 
>> being much 
>> worse the longer the BalancerMember timeout is set to -- and we have 
>> some very long running requests and thus need this timeout to 
>> be /very/ 
>> large).
>>
>> This is not the only ill effect we've seen when hitting Tomcat 5.5.24 
>> with concurrent requests in this manner, but it is a good 
>> place to start.
>>
>> As for the native connector, it just works here.  So why don't I just 
>> use it?  Well, first off, we have to support Tomcat on 
>> Windows (32 and 
>> 64-bit), Linux, Solaris, HPUX (PA-RISC and Itanium), and AIX.  We've 
>> been unable to get the connector built on some of these 
>> platforms and on 
>> some we can't get the resulting binary to function (specifically on 
>> AIX).  Further, we had some stability issues with the native 
>> connector 
>> in the past and had considered the Java connector the safest, if not 
>> fastest, option -- and to a degree given that everything is 
>> Java I still 
>> feel that's the case.  Finally, however, this connector should just 
>> plain work.  Tomcat shouldn't be a cripple unless/until you manage to 
>> build a native connector for your platform.
>>
>> Any troubleshooting and/or debugging ideas (e.g. where 
>> exactly to place 
>> breakpoints, what logs to turn on, etc, etc) would be 
>> /greatly/ appreciated.
>>
>> --
>> Jess Holle
>> jessh@ptc.com
>>
>> P.S. If this should go to the user's mailing list instead 
>> that's fine, 
>> but this really seemed like a developer-oriented issue to me.
>>
>>
>>     
>
>
>
> This message is intended only for the use of the person(s) listed above as the intended recipient(s), and may contain information that is PRIVILEGED and CONFIDENTIAL.  If you are not an intended recipient, you may not read, copy, or distribute this message or any attachment. If you received this communication in error, please notify us immediately by e-mail and then delete all copies of this message and any attachments.
>
> In addition you should be aware that ordinary (unencrypted) e-mail sent through the Internet is not secure. Do not send confidential or sensitive information, such as social security numbers, account numbers, personal identification numbers and passwords, to us via ordinary (unencrypted) e-mail.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: dev-help@tomcat.apache.org
>
>   


RE: Serious non-native AJP connector issue

Posted by Bill Barker <wb...@wilshire.com>.
Yes, since you know in advance how many connections you are going to get to
the AJP connector, you can configure it so that it has enough threads to
handle all of those connections.  That is why it doesn't attempt to handle
the case when the concurrency goes above maxThreads.

 

> -----Original Message-----
> From: Jess Holle [mailto:jessh@ptc.com] 
> Sent: Thursday, June 14, 2007 2:58 PM
> To: Tomcat Developers List
> Cc: Dobson, Simon; Wang, Andy; Fenlason, Josh
> Subject: Serious non-native AJP connector issue
> 
> We're facing a /serious /issue with the non-native AJP connector.
> 
> To put it most simply, some requests seem to "get lost" in Tomcat in 
> various cases involving concurrent requests -- and not 
> egregious numbers 
> of concurrent requests, either.
> 
> For instance,
> 
>    1. Use a Tomcat 5.5.23 with a configuration like:
> 
>         <Connector port="8010"
>                    minSpareThreads="4" maxSpareThreads="12"
>     maxThreads="18" acceptCount="282"
>                    tomcatAuthentication="false"
>     useBodyEncodingForURI="true" URIEncoding="UTF-8"
>                    enableLookups="false" redirectPort="8443"
>     protocol="AJP/1.3" />
> 
>     (which are intended solely for making it easier to test 
> concurrency
>     issues that to overrun a realistic 'maxThreads' setting) and as a
>     control, similar thread pool settings on the direct HTTP 
> connector:
> 
>         <Connector port="8080" maxHttpHeaderSize="8192"
>                    minSpareThreads="4" maxSpareThreads="12"
>     maxThreads="18" acceptCount="282"
>                    enableLookups="false" redirectPort="8443"
>                    connectionTimeout="20000" 
> disableUploadTimeout="true" />
> 
>    2. Use an Apache 2.2.4 with mod_proxy_ajp with a 
> configuration like:
> 
>         <Proxy balancer://ajpWorker>
>         BalancerMember ajp://localhost:8010 min=16 max=300 smax=40
>         ttl=900 keepalive=Off timeout=900
>         </Proxy>
> 
>         RewriteEngine on
>         RewriteRule ^(/TestApp/(.*\.jsp(.*)|servlet/.*|.*\.jar))$
>         balancer://ajpWorker$1 [P]
> 
>     (on Windows in this case; similar results can be obtained on Linux
>     at least)
> 
>    3. Use a simple test JSP page (placed in a web app 
> containing nothing
>       else):
> 
>         <%@page session="false"
>         %><%@page contentType="text/html" pageEncoding="UTF-8"
>         %><%!
>           private static final String  titleString = "Sleepy 
> Test JSP Page";
>         %><html>
>         <head>
>         <%
>           String  sleepSeconds = request.getParameter( "secs" );
>           if ( sleepSeconds == null )
>             sleepSeconds = "1";
>           long  secsToSleep = Long.parseLong( sleepSeconds );
>           Thread.sleep( 1000L * secsToSleep );
>         %>
>         <meta http-equiv="Content-Type" content="text/html; 
> charset=UTF-8"/>
>         <title><%=titleString%></title>
>         </head>
>         <body>
>         <center>
>         <h2><%=titleString%>: SUCCESS!</h2>
>         [Slept <%= secsToSleep %> seconds.]
>         </center>
>         </body>
>         </html>
> 
>    4. Hit the page with ab
>           * First, test direct Tomcat connections:
>                 o ab -n 24 -c 24 -A wcadmin:wcadmin
>                   http://hostname:*8080*/TestApp/test.jsp?secs=3
>                       + Result: All requests complete sucessfully in
>                         6118 ms.
>           * Second, test connections via Apache:
>                 o ab -n 24 -c 24 -A wcadmin:wcadmin
>                   http://hostname/TestApp/test.jsp?secs=3
>                       + Result: Only 17 requests complete before ab
>                         times out.
>           * Third, test connections via Apache again soon (under the
>             BalancerMember 'timeout' seconds) after the last test
>                 o ab -n 24 -c 24 -A wcadmin:wcadmin
>                   http://hostname/TestApp/test.jsp?secs=3
>                       + Result: Only 9 requests complete 
> before ab times
>                         out.
> 
> Something is clearly /horribly/ wrong with the handling of any 
> concurrency over 'maxThreads' in this case.  Even so, there 
> seems to be 
> some sort of "off-by-one" error in that only 17 requests 
> complete, not 
> 18.  Worse, this has a lingering effect that decreases 
> Tomcat's ability 
> to concurrent requests thereafter (with this impact seemingly 
> being much 
> worse the longer the BalancerMember timeout is set to -- and we have 
> some very long running requests and thus need this timeout to 
> be /very/ 
> large).
> 
> This is not the only ill effect we've seen when hitting Tomcat 5.5.24 
> with concurrent requests in this manner, but it is a good 
> place to start.
> 
> As for the native connector, it just works here.  So why don't I just 
> use it?  Well, first off, we have to support Tomcat on 
> Windows (32 and 
> 64-bit), Linux, Solaris, HPUX (PA-RISC and Itanium), and AIX.  We've 
> been unable to get the connector built on some of these 
> platforms and on 
> some we can't get the resulting binary to function (specifically on 
> AIX).  Further, we had some stability issues with the native 
> connector 
> in the past and had considered the Java connector the safest, if not 
> fastest, option -- and to a degree given that everything is 
> Java I still 
> feel that's the case.  Finally, however, this connector should just 
> plain work.  Tomcat shouldn't be a cripple unless/until you manage to 
> build a native connector for your platform.
> 
> Any troubleshooting and/or debugging ideas (e.g. where 
> exactly to place 
> breakpoints, what logs to turn on, etc, etc) would be 
> /greatly/ appreciated.
> 
> --
> Jess Holle
> jessh@ptc.com
> 
> P.S. If this should go to the user's mailing list instead 
> that's fine, 
> but this really seemed like a developer-oriented issue to me.
> 
> 



This message is intended only for the use of the person(s) listed above as the intended recipient(s), and may contain information that is PRIVILEGED and CONFIDENTIAL.  If you are not an intended recipient, you may not read, copy, or distribute this message or any attachment. If you received this communication in error, please notify us immediately by e-mail and then delete all copies of this message and any attachments.

In addition you should be aware that ordinary (unencrypted) e-mail sent through the Internet is not secure. Do not send confidential or sensitive information, such as social security numbers, account numbers, personal identification numbers and passwords, to us via ordinary (unencrypted) e-mail.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org