Buggy mod_jk2 AJP 13 communications?

[James Courtney <Jc...@inphonic.com>]

Developers,
	Forgive the post to the developer list but I honestly feel that my question(s) are detailed enought to be of interest to the development community for Tomcat and have thus far remained largely unaddressed by the tomcat-user community.  Please also forgive the length of the email but as a developer I try to provide any and all information that I've gathered to give anyone willing to help me as much relevant information as possible.  Please keep in mind while reading and formulating your answer(s) that I really don't have the luxury of taking wild stabs in the dark with this as it's a current production problem for us.  I need to know the best course of action to take to correct our systems to provide reliable service to our users.

We recently launched our application running on 2 Tomcat 4.1.24/JDK1.4.1/Sun 2.8/Sparc machines load balanced by an Apache 2.0.45/Sun2.8/Sparc machine using mod_jk2.  Throughout testing (including some basic load testing) we experienced very good behavior from our cluster and are in general very pleased with the performance over our previous system of Apache 1.3.19/Tomcat 3.2.2/mod_jk.  We built our mod_jk2 Apache module using the 4.1.24 Tomcat-Connectors source release provided with the Tomcat 4.1.24 release.  This seems to be slightly more recent than the last public release of mod_jk2 which I think was 2.0.2 according to CVS and what I've read.

Our problem is that we notice intermittent and not infrequent lapses in our application.  These appear as very slow performance or complete lack of connectivity to the Apache server but each of the Tomcat servers is functioning normally when we connect directly to those.

Apache serves no contect, static or dynamic, and everything is pushed to the Tomcat servers as the bulk of our page content is dynamic and we decided to keep the config simple at the expense of a little network performance on what static content we have.

It's about the peak time of day for us and we have 50-60 active TCP connections from our Apache server to EACH Tomcat server, all in the ESTABLISHED state according to netstat.  We have an additional 220 or so TCP connections from the internet to our Apache server of which about 30 are ESTABLISHED, about 20 are FIN_WAIT (and FIN_WAIT_2), and about 170 are TIME_WAIT.

Assuming ajp13 works like HTTP1.1 this all makes sense as the Apache would keep sockets open to the Tomcats and internet users opening and closing connections and browsers to the Apache would probably create a pattern like that above.

I've attached several of our config files for your reference:
1) httpd.conf (for Apache or course)     
2) workers2.properties (for mod_jk2 of course)
3) server.xml (from one Tomcat, both are indentical with exception of jvmRoute)
4) jk2.properties (Still don't know the point of this but here it is)
3) apache.info (a dump using httpd -V)


I have several general questions which I feel can only be answered by those fairly familiar with the mod_jk/jk2 code.

1) Which is the preferred connector at this time in terms of reliability and scalability.
2) What is the preferred version/release of that connector.
3) Should both the java and c/apache side of the connector be built and installed together onto Tomcat 4.1.24 for compatibility or is the c/apache side alone sufficient to work reliably with the CoyoteConnector/JkCoyoteHandler packaged with the 4.1.24 build?
4) Does Apache mod_jk(2) pool a set of connections to Tomcat not to exceed the maxProcessors parameter of the Tomcat CoyoteConnector?  Could my connections be timing out?  Should I set connectionTimeout to -1 (it's currently 0)?
5) What parameters affect mod_jk(2) performance.  I see from the code that the Channel.Socket (as of the TOMCAT_4_1_24 CVS tag) will accept settings for timeout, keepalive, and nodelay but that these are not documented.  Could these help?
6) Could my Apache 2.0.45 just be foobar?  This seems unlikely, the config is pretty straight forward.

I'll do my best to answer any follow up questions quickly and accurately and to gather any information required.  Thank you all for your help!


Best Regards,
	Jamey



James Courtney
InPhonic, Inc.
Hayward, CA
 <<apache.info>>  <<httpd.conf>>  <<jk2.properties>>  <<server.xml>>  <<workers2.properties>> 

Re: Buggy mod_jk2 AJP 13 communications?

[Henri Gomez <hg...@apache.org>]

James Courtney a écrit :

> Developers,
> 	Forgive the post to the developer list but I honestly feel that my question(s) are detailed enought to be of interest to the development community for Tomcat and have thus far remained largely unaddressed by the tomcat-user community.  Please also forgive the length of the email but as a developer I try to provide any and all information that I've gathered to give anyone willing to help me as much relevant information as possible.  Please keep in mind while reading and formulating your answer(s) that I really don't have the luxury of taking wild stabs in the dark with this as it's a current production problem for us.  I need to know the best course of action to take to correct our systems to provide reliable service to our users.
> 
> We recently launched our application running on 2 Tomcat 4.1.24/JDK1.4.1/Sun 2.8/Sparc machines load balanced by an Apache 2.0.45/Sun2.8/Sparc machine using mod_jk2.  Throughout testing (including some basic load testing) we experienced very good behavior from our cluster and are in general very pleased with the performance over our previous system of Apache 1.3.19/Tomcat 3.2.2/mod_jk.  We built our mod_jk2 Apache module using the 4.1.24 Tomcat-Connectors source release provided with the Tomcat 4.1.24 release.  This seems to be slightly more recent than the last public release of mod_jk2 which I think was 2.0.2 according to CVS and what I've read.
> 
> Our problem is that we notice intermittent and not infrequent lapses in our application.  These appear as very slow performance or complete lack of connectivity to the Apache server but each of the Tomcat servers is functioning normally when we connect directly to those.
> 
> Apache serves no contect, static or dynamic, and everything is pushed to the Tomcat servers as the bulk of our page content is dynamic and we decided to keep the config simple at the expense of a little network performance on what static content we have.
> 
> It's about the peak time of day for us and we have 50-60 active TCP connections from our Apache server to EACH Tomcat server, all in the ESTABLISHED state according to netstat.  We have an additional 220 or so TCP connections from the internet to our Apache server of which about 30 are ESTABLISHED, about 20 are FIN_WAIT (and FIN_WAIT_2), and about 170 are TIME_WAIT.
> 
> Assuming ajp13 works like HTTP1.1 this all makes sense as the Apache would keep sockets open to the Tomcats and internet users opening and closing connections and browsers to the Apache would probably create a pattern like that above.
> 
> I've attached several of our config files for your reference:
> 1) httpd.conf (for Apache or course)     
> 2) workers2.properties (for mod_jk2 of course)
> 3) server.xml (from one Tomcat, both are indentical with exception of jvmRoute)
> 4) jk2.properties (Still don't know the point of this but here it is)
> 3) apache.info (a dump using httpd -V)
> 
> 
> I have several general questions which I feel can only be answered by those fairly familiar with the mod_jk/jk2 code.
> 
> 1) Which is the preferred connector at this time in terms of reliability and scalability.

mod_jk is the preferred in term of reliability since it's older and more 
tested, but jk2 is the future.

> 2) What is the preferred version/release of that connector.

mod_jk 1.2.5 should be release soon.

> 3) Should both the java and c/apache side of the connector be built and installed together onto Tomcat 4.1.24 for compatibility or is the c/apache side alone sufficient to work reliably with the CoyoteConnector/JkCoyoteHandler packaged with the 4.1.24 build?

mod_jk should be built from jakarta-tomcat-connectors release.

> 4) Does Apache mod_jk(2) pool a set of connections to Tomcat not to exceed the maxProcessors parameter of the Tomcat CoyoteConnector?  Could my connections be timing out?  Should I set connectionTimeout to -1 (it's currently 0)?

The rule is that the MAX NUMBERS of HTTPD connexions should be less than 
the maxProcessors in ajp13. For instance if you have 2 tomcat with each 
150 maxProcessors, you should restrict Apache HTTPD to have not more 
than 250 connexions

> 5) What parameters affect mod_jk(2) performance.  I see from the code that the Channel.Socket (as of the TOMCAT_4_1_24 CVS tag) will accept settings for timeout, keepalive, 
and nodelay but that these are not documented.  Could these help?

See the nodelay (NAAGLE related)

> 6) Could my Apache 2.0.45 just be foobar?  This seems unlikely, the config is pretty straight forward.

I'm using Apache 2.0.46 with Tomcats (3.3.1a) and mod_jk 1.2.4 and have 
no problems with it. In my situation Apache server static contents like 
images and css.

Regards


---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-dev-help@jakarta.apache.org