You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Fred Whipple <fw...@imagineis.com> on 2003/08/25 17:54:38 UTC
mod_jk and Tomcat Lose Touch
Hi all,
On one of our servers, we were running Red Hat Linux 7.3 with Apache
1.3.X, mod_jk, Sun JDK 1.4.1, and many (~150) instances Tomcat of
various flavors. Everything worked perfectly until we upgraded to Red
Hat 9. Now, it seems like mod_jk and Tomcat are losing touch until the
Tomcat JVM is restarted. This happens after maybe a day or so using
both Apache 1.3.X and Apache 2.0.X.
RE: mod_jk and Tomcat Lose Touch
Posted by Fred Whipple <fw...@imagineis.com>.
This is what I thought, too, originally. However it wouldn't make sense
since very few times do resources become *fewer* by default as software
versions increase.
With that in mind, and a hunch up my sleeve, I actually did some interesting
testing. My test platform is my notebook, running RHL 9, 512MB RAM, Apache
1.3.27, JDK 1.4.1_03, and Tomcat 4.1.24. It's the exact same version of all
software, including the kernel, that we're running on the server that's
experiencing the problem.
What I did was create 200 virtual hosts, 200 Tomcat instances running on 200
different ports, and tied them all together with mod_jk. I then created a
test script which used 'lynx' to test a simple JSP on each virtual host and
display the which JVM it was testing and what its results were.
My results were *very* interesting. They are *reproducable* and
*predictable*, which is at least better news than completely random!
On the same software platform, where all defaults and limitations were
exactly the same, the major affecting factors seem to be memory and swap,
*not* any given resource within the OS itself.
On the production system, this problem will not appear until it's running,
say, 100 JVM's or so. The server has 2GB of RAM, 4GB of swap. On my
notebook, however, it will appear with only 40 JVM's, which has 512MB of RAM
and 1GB swap. Of course, when it's starting JVM's 30-40, it's doing a lot
of swapping as resources are allocated. So how it turns out is as follows:
If I start 1 JVM per minute, up to 40, it does just a little swapping per
instance as it starts up. All JVM's up to #25 or so will work perfectly.
Once you startup #26 and up, you start to lose a JVM here and there --
mod_jk will report its error and you'll never get a response. Essentially,
mod_jk and the JVM lose touch. This will continue up to JVM #40 or so.
Starting 1 per minute, I lost maybe 3-4 JVM's along the way. If I then
start them back up, all the JVM's will work for a while, but eventually one
or two will drop out again. Any more than 40 JVM's, and they all start
dropping like flys.
If I start all 40 JVM's 10 seconds apart, the swapping is tremendous by JVM
#25, and *no* JVM will work at all after #25. Also, all the first 25 drop
out as well.
All of this behavior seems consistent with the behavior on the server, as
well, only scaled-down. The server has 4x the RAM, so it doesn't exhibit
this behavior until its memory gets full and it starts swapping processes
out. I.e., if I start 100 JVM's on the server, it doesn't use any (or
hardly any) swap space, and everything works perfectly. As the Web
applications initiliaze, sessions are built, etc. eventually the server
levels out at ~2GB of swap space being used, and this is when the problems
start. This leads me to believe that the problem is with either memory not
being properly allocated to mod_jk, even though there's plenty of swap
available, *or* a timing issue where it tries to send a request to a JVM but
for a split-second it doesn't get a response (due to paging, or whatever)
and then perminently loses touch. This exact situation occurred a long time
ago back in the days of JServ and mod_jserv.
This is probably an issue for the developers to look at -- are there any
mod_jk developers on this list? :-) I'm sure their knee-jerk reaction would
be "upgrade to mod_jk2 and Apache2," but this is not a practical solution
for me.
Thanks for any help, and for reading so far!
-Fred
> Sounds to me like an OS resource issue. Total number of allowed open
> connections, etc. Since it worked on 7.x and not on 9, I
> would consider
> that a default changed somewhere, somehow.
>
> John
Re: mod_jk and Tomcat Lose Touch
Posted by John Turner <to...@johnturner.com>.
Sounds to me like an OS resource issue. Total number of allowed open
connections, etc. Since it worked on 7.x and not on 9, I would consider
that a default changed somewhere, somehow.
John
> Also, the same Apache 1.3.X and Tomcat 3.X and 4.X configurations worked
> perfectly on Red Hat 7.3, it was not until RH 9 that this began occuring.
> The behavior is consistent with Apache 1.3 and Apache 2 on RH 9. I've tried
> re-compiling and all the usual tricks.
>
> ...
>
>
>>Do you use multiple Apaches on the server?
>
RE: mod_jk and Tomcat Lose Touch
Posted by Fred Whipple <fw...@imagineis.com>.
> Looks like there is a mismatch between the workers.properties and the
> server.xml (Tomcat's config file). The server.xml sets up Tomcat. More
> importantly, it sets up a Connector which opens a socket
> connection on a
> defined port (Look for the Connector' section in the Service
> section of
> the server.xml). This port is what mod_jk needs to know about in the
> workers.properties. Some snippets from my config:
...
> Check out both of these files and make sure the config is setup
> cporrectly. It looks like either the Connector is not
> configured or the
> workers.properties does not have the right info for the
> workers spawned
> by Apache.
Unfortunately it's not nearly this simple :-( Per my original post, we have
~150 Tomcat instances running, all properly configured, and 1 instance of
Apache on the server. If I say, 'killall' the JVM's, and then start them
back up again, everything will work perfectly. After a day or two, some of
the sites will then lose their connectivity to Apache/JK. Simply restarting
the JVM will "fix" things for a further day or two (for the sites configured
to work with the JVM that I restart), thereby ruling-out a misconfiguration
(though not ruling-out a possible non-optimal-configuration).
Also, the same Apache 1.3.X and Tomcat 3.X and 4.X configurations worked
perfectly on Red Hat 7.3, it was not until RH 9 that this began occuring.
The behavior is consistent with Apache 1.3 and Apache 2 on RH 9. I've tried
re-compiling and all the usual tricks.
...
> Do you use multiple Apaches on the server?
Just one.
> Since I would be speculating wildly, why not post the
> workers.properties
> file as well as the Connector tags in the server.xml on the Tomcats.
Well, 150 workers in the workers.properties file would be a bit large for
starters ;-) So would 150 server.xml's... But regardless, I'm certain
they're all configured properly, it's just a matter of Tomcat's AJP 1.3
service and mod_jk losing touch after some period of time.
-Fred
RE: mod_jk and Tomcat Lose Touch
Posted by Ben Ricker <br...@wellinx.com>.
On Tue, 2003-08-26 at 15:33, Fred Whipple wrote:
> > In the settings you put in httpd.conf, you should have added a
> > 'LogLevel' directive. If you did not, then you can! The line
> > I have is:
> >
> > JkLogLevel error
> >
> > Change the 'error' to debug and restart Apache.
>
> This worked well, and yielded some errors:
>
> **********
> [Tue Aug 26 16:25:19 2003] [jk_connect.c (203)]: jk_open_socket, connect()
> failed errno = 111
> [Tue Aug 26 16:25:19 2003] [jk_ajp_common.c (626)]: Error connecting to
> tomcat. Tomcat is probably not started or is listenning on the wrong port.
> Failed errno = 111
Looks like there is a mismatch between the workers.properties and the
server.xml (Tomcat's config file). The server.xml sets up Tomcat. More
importantly, it sets up a Connector which opens a socket connection on a
defined port (Look for the Connector' section in the Service section of
the server.xml). This port is what mod_jk needs to know about in the
workers.properties. Some snippets from my config:
-----Server.xml snippet-----
<!-- Define an AJP 1.3 Connector on port 11009 -->
<Connector className="org.apache.ajp.tomcat4.Ajp13Connector"
address="10.1.1.1" port="11009" minProcessors="20" maxProcessors="80"
acceptCount="10" debug="0"/>
-----End Server.xml snippet-----
-----mod_jk's workers.properties file snippet-----
# ------------------------
# First tomcat server
# ------------------------
worker.tomcat1.port=11009
worker.tomcat1.host=10.1.1.1
worker.tomcat1.type=ajp13
-----End mod_jk's workers.properties file snippet-----
Check out both of these files and make sure the config is setup
cporrectly. It looks like either the Connector is not configured or the
workers.properties does not have the right info for the workers spawned
by Apache.
> All of these errors appeared at the same time I was trying to view a JSP on
> a site that had "lost touch." I was doing a tail -f and the log file had
> been quiet until I tried to view this JSP -- then all these popped-up at
> about the same time.
>
> Immidiately I opened a telnet session to this particular Tomcat instance's
> AJP 1.3 port, and verified it was responding. Also, *all* of the Tomcat
> instances are on the same host as the Web server, so all the workers are
> traveling over 127.0.0.1.
Do you use multiple Apaches on the server?
Since I would be speculating wildly, why not post the workers.properties
file as well as the Connector tags in the server.xml on the Tomcats.
Ben Ricker
Wellinx.com
RE: mod_jk and Tomcat Lose Touch
Posted by Fred Whipple <fw...@imagineis.com>.
> In the settings you put in httpd.conf, you should have added a
> 'LogLevel' directive. If you did not, then you can! The line
> I have is:
>
> JkLogLevel error
>
> Change the 'error' to debug and restart Apache.
This worked well, and yielded some errors:
**********
[Tue Aug 26 16:25:19 2003] [jk_connect.c (203)]: jk_open_socket, connect()
failed errno = 111
[Tue Aug 26 16:25:19 2003] [jk_ajp_common.c (626)]: Error connecting to
tomcat. Tomcat is probably not started or is listenning on the wrong port.
Failed errno = 111
[Tue Aug 26 16:25:19 2003] [jk_ajp_common.c (874)]: Error connecting to the
Tomcat process.
[Tue Aug 26 16:25:19 2003] [jk_ajp_common.c (1190)]: sending request to
tomcat failed in send loop. err=0
[Tue Aug 26 16:25:19 2003] [jk_connect.c (203)]: jk_open_socket, connect()
failed errno = 111
[Tue Aug 26 16:25:19 2003] [jk_ajp_common.c (626)]: Error connecting to
tomcat. Tomcat is probably not started or is listenning on the wrong port.
Failed errno = 111
[Tue Aug 26 16:25:19 2003] [jk_ajp_common.c (874)]: Error connecting to the
Tomcat process.
[Tue Aug 26 16:25:19 2003] [jk_ajp_common.c (1190)]: sending request to
tomcat failed in send loop. err=1
[Tue Aug 26 16:25:19 2003] [jk_connect.c (203)]: jk_open_socket, connect()
failed errno = 111
[Tue Aug 26 16:25:19 2003] [jk_ajp_common.c (626)]: Error connecting to
tomcat. Tomcat is probably not started or is listenning on the wrong port.
Failed errno = 111
[Tue Aug 26 16:25:19 2003] [jk_ajp_common.c (874)]: Error connecting to the
Tomcat process.
[Tue Aug 26 16:25:19 2003] [jk_ajp_common.c (1190)]: sending request to
tomcat failed in send loop. err=2
[Tue Aug 26 16:25:19 2003] [jk_ajp_common.c (1198)]: Error connecting to
tomcat. Tomcat is probably not started or is listenning on the wrong port.
Failed errno = 111
[Tue Aug 26 16:25:19 2003] [jk_ajp_common.c (970)]: ERROR sending data to
client. Connection aborted or network problems
[Tue Aug 26 16:25:50 2003] [jk_ajp_common.c (970)]: ERROR sending data to
client. Connection aborted or network problems
[Tue Aug 26 16:26:04 2003] [jk_ajp_common.c (970)]: ERROR sending data to
client. Connection aborted or network problems
[Tue Aug 26 16:26:15 2003] [jk_ajp_common.c (970)]: ERROR sending data to
client. Connection aborted or network problems
[Tue Aug 26 16:26:22 2003] [jk_ajp_common.c (970)]: ERROR sending data to
client. Connection aborted or network problems
**********
All of these errors appeared at the same time I was trying to view a JSP on
a site that had "lost touch." I was doing a tail -f and the log file had
been quiet until I tried to view this JSP -- then all these popped-up at
about the same time.
Immidiately I opened a telnet session to this particular Tomcat instance's
AJP 1.3 port, and verified it was responding. Also, *all* of the Tomcat
instances are on the same host as the Web server, so all the workers are
traveling over 127.0.0.1.
Any ideas?
Thanks!
-Fred
RE: mod_jk and Tomcat Lose Touch
Posted by Ben Ricker <br...@wellinx.com>.
On Mon, 2003-08-25 at 13:08, Fred Whipple wrote:
> Ah, the symptoms, of course ;-)
>
> There actually aren't any errors AFAIK. Perhaps there's a more verbose
> logging level? I couldn't find any obvious info on this in the manual.
In the settings you put in httpd.conf, you should have added a
'LogLevel' directive. If you did not, then you can! The line I have is:
JkLogLevel error
Change the 'error' to debug and restart Apache.
Ben Ricker
Wellinx, Inc.
> What actually happens is if I make a request to a JSP, for example, that
> should normally be forwarded onto Tomcat, the request simply hangs forever.
> Apache logs the request nothing in the Tomcat logs suggest to me it's
> receiving the request.
>
> Thanks,
>
> -Fred
>
> > -----Original Message-----
> > From: Ben Ricker [mailto:bricker@wellinx.com]
> > Sent: Monday, August 25, 2003 1:09 PM
> > To: tomcat-user@jakarta.apache.org
> > Subject: Re: mod_jk and Tomcat Lose Touch
> >
> >
> > What kind of error are you getting that makes you thing they
> > are "losing
> > touch". Check the mod_jk log and see what kind of errors it
> > is throwing
> > and post an example.
> >
> > Ben Ricker
> > Wellinx, Inc.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
>
>
RE: mod_jk and Tomcat Lose Touch
Posted by Fred Whipple <fw...@imagineis.com>.
Ah, the symptoms, of course ;-)
There actually aren't any errors AFAIK. Perhaps there's a more verbose
logging level? I couldn't find any obvious info on this in the manual.
What actually happens is if I make a request to a JSP, for example, that
should normally be forwarded onto Tomcat, the request simply hangs forever.
Apache logs the request nothing in the Tomcat logs suggest to me it's
receiving the request.
Thanks,
-Fred
> -----Original Message-----
> From: Ben Ricker [mailto:bricker@wellinx.com]
> Sent: Monday, August 25, 2003 1:09 PM
> To: tomcat-user@jakarta.apache.org
> Subject: Re: mod_jk and Tomcat Lose Touch
>
>
> What kind of error are you getting that makes you thing they
> are "losing
> touch". Check the mod_jk log and see what kind of errors it
> is throwing
> and post an example.
>
> Ben Ricker
> Wellinx, Inc.
Re: mod_jk and Tomcat Lose Touch
Posted by Ben Ricker <br...@wellinx.com>.
What kind of error are you getting that makes you thing they are "losing
touch". Check the mod_jk log and see what kind of errors it is throwing
and post an example.
Ben Ricker
Wellinx, Inc.
On Mon, 2003-08-25 at 10:54, Fred Whipple wrote:
> Hi all,
>
> On one of our servers, we were running Red Hat Linux 7.3 with Apache
> 1.3.X, mod_jk, Sun JDK 1.4.1, and many (~150) instances Tomcat of
> various flavors. Everything worked perfectly until we upgraded to Red
> Hat 9. Now, it seems like mod_jk and Tomcat are losing touch until the
> Tomcat JVM is restarted. This happens after maybe a day or so using
> both Apache 1.3.X and Apache 2.0.X.
>
> >From what we've observed, it seems to a problem specific to mod_jk.
> This is because running Apache 1.3.X or 2.0.X under Red Hat 9 with the
> old-skool mod_jserv and ~100 Tomcat instances does not cause a problem
> at all. Also, running either Apache 2 or 1.3 on RH 9 with just a couple
> JVM's does not cause a problem either.
>
> I've already tried re-compiling mod_jk on RH9, the same server that's
> having the trouble to account for any library change issues. No dice
> I'm afraid.
>
> Any ideas?
>
> Thanks!
>
> -Fred
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
>
>