You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Nick Williams <ni...@puresafety.com> on 2011/05/21 01:18:46 UTC
JK Connector failure after IIS recycle - version 1.2.30
Environment:
Windows Server 2008
IIS 7.0
Tomcat 6.0.29
ISAPI Redirect JK Connector 1.2.30
At 14:06:57 this afternoon, IIS performed a recycle (it does this ever 29
hours and has for years without causing us problems):
“A worker process with process id of '10536' serving application pool
'DefaultAppPool' has requested a recycle because the worker process reached
its allowed processing time limit.”
The last log entry in the IIS log file until the recycle complete was at
14:06:57. We get about 100 requests per second, and there were about 100
requests at 14:06:57.
>From 14:06:58-14:07:01 (the three seconds following the recycle), the
following log messages appeared in the JK connector ISAPI redirect log (more
information following this abbreviated log output):
[Fri May 20 14:06:23.707 2011] [10536:7896] [error]
ajp_service::jk_ajp_common.c (2559): (s01aspgrp03) connecting to tomcat
failed.
[Fri May 20 14:06:58.745 2011] [12064:6756] [error]
wc_create_worker::jk_worker.c (151): factory for lb failed for lbaspgrp10
[Fri May 20 14:06:58.854 2011] [12064:6756] [error]
build_worker_map::jk_worker.c (262): failed to create worker lbaspgrp10
[Fri May 20 14:06:58.854 2011] [12064:6756] [error]
uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with
name 'lbaspgrp14' in uri map post processing.
… (hundreds of these messages) …
[Fri May 20 14:06:59.386 2011] [12064:6756] [error]
uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with
name 'lbaspgrp13' in uri map post processing.
[Fri May 20 14:06:59.433 2011] [12064:8708] [error]
ajp_worker_factory::jk_ajp_common.c (2929): allocating ajp worker record
from shared memory
[Fri May 20 14:06:59.433 2011] [12064:8708] [error]
wc_create_worker::jk_worker.c (151): factory for ajp13 failed for default
[Fri May 20 14:06:59.433 2011] [12064:8708] [error]
build_worker_map::jk_worker.c (262): failed to create worker default
[Fri May 20 14:06:59.433 2011] [12064:8708] [error]
uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with
name 'lbaspgrp14' in uri map post processing.
… (hundreds of these messages) …
[Fri May 20 14:07:00.012 2011] [12064:8708] [error]
uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with
name 'lbaspgrp13' in uri map post processing.
[Fri May 20 14:07:00.043 2011] [12064:14296] [error]
ajp_worker_factory::jk_ajp_common.c (2929): allocating ajp worker record
from shared memory
[Fri May 20 14:07:00.043 2011] [12064:14296] [error]
wc_create_worker::jk_worker.c (151): factory for ajp13 failed for default
[Fri May 20 14:07:00.043 2011] [12064:14296] [error]
build_worker_map::jk_worker.c (262): failed to create worker default
[Fri May 20 14:07:00.043 2011] [12064:14296] [error]
uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with
name 'lbaspgrp14' in uri map post processing.
… (hundreds of these messages) …
[Fri May 20 14:07:00.606 2011] [12064:14296] [error]
uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with
name 'lbaspgrp13' in uri map post processing.
[Fri May 20 14:07:00.638 2011] [12064:14904] [error]
ajp_worker_factory::jk_ajp_common.c (2929): allocating ajp worker record
from shared memory
[Fri May 20 14:07:00.638 2011] [12064:14904] [error]
wc_create_worker::jk_worker.c (151): factory for ajp13 failed for default
[Fri May 20 14:07:00.653 2011] [12064:14904] [error]
build_worker_map::jk_worker.c (262): failed to create worker default
[Fri May 20 14:07:00.653 2011] [12064:14904] [error]
uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with
name 'lbaspgrp14' in uri map post processing.
… (hundreds of these messages) …
[Fri May 20 14:07:01.169 2011] [12064:14904] [error]
uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with
name 'lbaspgrp13' in uri map post processing.
Beginning at 14:06:59 and continuing until we shut down IIS and failed over
to another server 30 minutes later, requests began appearing in the IIS log
file again, but at this time ALL requests were replied to with a 500
internal server error. We have been running this server with this
configuration for over a year without any issues like this until now. No
configuration settings were changed. No workers were added. We do not have
shm_size set anywhere since, per the documentation, that directive is not
needed anymore as of 1.2.27.
I have been unable to find any useful information about this elsewhere, save
this bug:
https://issues.apache.org/bugzilla/show_bug.cgi?id=40877
However, that bug was apparently fixed back in 2007 with version 1.2.20. So,
I’m not sure if it’s related or not.
Any insights? Has anyone else seen this? Is this an IIS problem or a
new/reoccurring JK Connector bug or a configuration problem?
Thanks in advance!
Nick
Re: JK Connector failure after IIS recycle - version 1.2.30
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
André,
On 5/26/2011 4:19 AM, André Warnier wrote:
> 1) it happened only once
> 2) the same configuration has been working without showing such a
> problem for at least one year
> 3) neither IIS nor Tomcat nor isapi_redirector were recently updated
> 4) the usage pattern for the website did not change significantly in a
> way that could explain a problem
> 5) there was no external network or system issue which could explain the
> problem
> 6) the IIS logs do not show the problem to be at the IIS level, and the
> Tomcat logs do not show the problem to be at the Tomcat level
7) the server and network hardware is still trustworthy
That last one is always subject to change without notice :)
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk3eXAYACgkQ9CaO5/Lv0PANBQCdEhqvhsn9Uhh3NXgKuvbX3Q+x
wXYAnj7w6vMkZ0ogq/AoHS+x17EGKKh1
=76pY
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: JK Connector failure after IIS recycle - version 1.2.30
Posted by André Warnier <aw...@ice-sa.com>.
Nick Williams wrote:
> Thanks for the insight. I'll give it a little more time, but I'm being
> pushed by my superiors here for an answer that I can't give, so I'll have
> to file a bug before long.
>
> Does anyone know if there are any other (open source OR commercial/paid)
> alternatives to integrating Tomcat with IIS and/or Apache?
>
Yes, there are other possibilities for both IIS and Apache (see below).
But before you look at that, you may want to investigate your particular issue a bit further.
Both mod_jk (for Apache) and isapi_redirector (for IIS) are and have been used by *lots*
of people for several years. This is not to say that there cannot be a bug in them, but
that at least they are generally pretty reliable and have stood a lot of testing and usage
in production. So before you switch to something else, you'd have to ask yourself how
well-tested these alternatives are. And as far as I know, most of them are "younger" than
mod_jk and isapi_redirector.
The second part is that from your description, the problem you mentioned has some
characteristics which make me a bit doubtful that the issue /originated/ with
isapi_redirector. Namely, as I understand your description :
1) it happened only once
2) the same configuration has been working without showing such a problem for at least one
year
3) neither IIS nor Tomcat nor isapi_redirector were recently updated
4) the usage pattern for the website did not change significantly in a way that could
explain a problem
5) there was no external network or system issue which could explain the problem
6) the IIS logs do not show the problem to be at the IIS level, and the Tomcat logs do not
show the problem to be at the Tomcat level
Now, as the old joke goes, pick any 5 of the 6 above; because in the way my mind works,
not all 6 can be true at the same time.
About the alternatives :
Both isapi_redirector and mod_jk use the AJP protocol, and the AJP connector at the Tomcat
level, to handle the connection between the front-end webserver and the back-end Tomcat.
Both isapi_redirector and mod_jk, in addition to being just a "proxying" solution,
incorporate a load-balancing aspect, and a failover aspect.
Also both are open-source and free.
Most other solutions below use the HTTP protocol and the HTTP connector at the Tomcat
level. Some of them are open-source and free, others not. Some of them offer
load-balancing and/or failover, others not.
isapi_redirector and mod_jk offer a lot of tuning capabilities; as far as I know, the
other solutions below offer a lot less such capabilities. Whether that is an advantage or
an inconvenient is in the eyes of the beholder.
1) IIS 7.0 has a built-in HTTP proxy capability. There is some configuration needed for
that at the IIS level, but I do not remember the details. As far as i remember, this does
not offer load-balancing or failover.
2) There are several IIS add-on modules available (not free nor open-source) which provide
such HTTP proxying capabilities for IIS. One of them is "isapi rewrite 3"
(http://www.helicontech.com/isapi_rewrite/).
3) For Apache, you can use the mod_proxy module with the mod_proxy_http sub-module, to
proxy requests from Apache to Tomcat via HTTP. This is open-source and free, and may
provide for load-balancing and maybe also failover, I don't remember.
4) For Apache, you can also use mod_proxy with the mod_proxy_ajp sub-module. This still
uses an AJP connection and the Tomcat AJP connector, and includes load-balancing and
failover as I recall. mod_proxy_ajp, compared to mod_jk, is a relative "new kid on the
block", and I really do not know how they compare in terms of performance or reliability.
My own experience is mainly with the Apache/mod_jk/Tomcat configuration, and I have never
had serious problems with it. I run several servers, but none of them has the kind of
load which you indicate for yours (100+ requests/second).
I have occasionally used 1, 2 and 3 above, so I know that they basically work; but also
not in very demanding circumstances. I have never used (4) above.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: JK Connector failure after IIS recycle - version 1.2.30
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Nick,
On 5/25/2011 6:31 PM, Nick Williams wrote:
> Thanks for the insight. I'll give it a little more time, but I'm being
> pushed by my superiors here for an answer that I can't give, so I'll have
> to file a bug before long.
Certainly when logging your bug, attach as much of the mod_jk log as you
can, and any other information that may be relevant (full version
numbers of everything involved, web server access log, etc.).
You did a pretty good job with your post to this list, so I suspect
you'll file a relatively complete bug report.
Sometimes posts just get lost in the shuffle. Bugs generally aren't
forgotten :)
> Does anyone know if there are any other (open source OR commercial/paid)
> alternatives to integrating Tomcat with IIS and/or Apache?
If you are using Apache httpd 2.2 or later, you should be able to use
mod_proxy_ajp which uses the same protocol as mod_jk but using
mod_proxy-style configuration and a completely different code base.
You could also switch-over to using mod_proxy_http and dump AJP altogether.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk3eW6sACgkQ9CaO5/Lv0PCj9QCfQmBMbTb43KK9G00g+TsDIXAc
C94AoMCQ8jKn8o77vA6w+xg+M6AqUDfM
=q4mP
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: JK Connector failure after IIS recycle - version 1.2.30
Posted by Nick Williams <ni...@puresafety.com>.
Thanks for the insight. I'll give it a little more time, but I'm being
pushed by my superiors here for an answer that I can't give, so I'll have
to file a bug before long.
Does anyone know if there are any other (open source OR commercial/paid)
alternatives to integrating Tomcat with IIS and/or Apache?
N
-----Original Message-----
From: André Warnier [mailto:aw@ice-sa.com]
Sent: Wednesday, May 25, 2011 3:49 PM
To: Tomcat Users List
Subject: Re: JK Connector failure after IIS recycle - version 1.2.30
Nick Williams wrote:
> Does anyone have any feeback? Do I need to report a bug?
>
My own experience with this list, is that when someone reports an issue or
asks a question
which fits with the knowledge or experience of the people on the list,
usually the
reaction time is short. So the fact that nobody has answered within the
last 3 working
days is unusual, and may be just an indication that nobody has a clue.
On the other hand, it may just mean that none of the relatively few people
qualified to
answer has been around yet, or has seen your original post.
About the bug report : I suppose you could, but 3 working days since the
initial problem
report may be a bit premature for an issue which, by your own description,
sounds for now
like a one-off and difficult for you or someone else to reproduce.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: JK Connector failure after IIS recycle - version 1.2.30
Posted by André Warnier <aw...@ice-sa.com>.
Nick Williams wrote:
> Does anyone have any feeback? Do I need to report a bug?
>
My own experience with this list, is that when someone reports an issue or asks a question
which fits with the knowledge or experience of the people on the list, usually the
reaction time is short. So the fact that nobody has answered within the last 3 working
days is unusual, and may be just an indication that nobody has a clue.
On the other hand, it may just mean that none of the relatively few people qualified to
answer has been around yet, or has seen your original post.
About the bug report : I suppose you could, but 3 working days since the initial problem
report may be a bit premature for an issue which, by your own description, sounds for now
like a one-off and difficult for you or someone else to reproduce.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: JK Connector failure after IIS recycle - version 1.2.30
Posted by Nick Williams <ni...@puresafety.com>.
Does anyone have any feeback? Do I need to report a bug?
Nick
*From:* Nick Williams [mailto:nicholas.williams@puresafety.com]
*Sent:* Friday, May 20, 2011 6:19 PM
*To:* 'Tomcat Users List'
*Subject:* JK Connector failure after IIS recycle - version 1.2.30
Environment:
Windows Server 2008
IIS 7.0
Tomcat 6.0.29
ISAPI Redirect JK Connector 1.2.30
At 14:06:57 this afternoon, IIS performed a recycle (it does this ever 29
hours and has for years without causing us problems):
“A worker process with process id of '10536' serving application pool
'DefaultAppPool' has requested a recycle because the worker process reached
its allowed processing time limit.”
The last log entry in the IIS log file until the recycle complete was at
14:06:57. We get about 100 requests per second, and there were about 100
requests at 14:06:57.
>From 14:06:58-14:07:01 (the three seconds following the recycle), the
following log messages appeared in the JK connector ISAPI redirect log (more
information following this abbreviated log output):
[Fri May 20 14:06:23.707 2011] [10536:7896] [error]
ajp_service::jk_ajp_common.c (2559): (s01aspgrp03) connecting to tomcat
failed.
[Fri May 20 14:06:58.745 2011] [12064:6756] [error]
wc_create_worker::jk_worker.c (151): factory for lb failed for lbaspgrp10
[Fri May 20 14:06:58.854 2011] [12064:6756] [error]
build_worker_map::jk_worker.c (262): failed to create worker lbaspgrp10
[Fri May 20 14:06:58.854 2011] [12064:6756] [error]
uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with
name 'lbaspgrp14' in uri map post processing.
… (hundreds of these messages) …
[Fri May 20 14:06:59.386 2011] [12064:6756] [error]
uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with
name 'lbaspgrp13' in uri map post processing.
[Fri May 20 14:06:59.433 2011] [12064:8708] [error]
ajp_worker_factory::jk_ajp_common.c (2929): allocating ajp worker record
from shared memory
[Fri May 20 14:06:59.433 2011] [12064:8708] [error]
wc_create_worker::jk_worker.c (151): factory for ajp13 failed for default
[Fri May 20 14:06:59.433 2011] [12064:8708] [error]
build_worker_map::jk_worker.c (262): failed to create worker default
[Fri May 20 14:06:59.433 2011] [12064:8708] [error]
uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with
name 'lbaspgrp14' in uri map post processing.
… (hundreds of these messages) …
[Fri May 20 14:07:00.012 2011] [12064:8708] [error]
uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with
name 'lbaspgrp13' in uri map post processing.
[Fri May 20 14:07:00.043 2011] [12064:14296] [error]
ajp_worker_factory::jk_ajp_common.c (2929): allocating ajp worker record
from shared memory
[Fri May 20 14:07:00.043 2011] [12064:14296] [error]
wc_create_worker::jk_worker.c (151): factory for ajp13 failed for default
[Fri May 20 14:07:00.043 2011] [12064:14296] [error]
build_worker_map::jk_worker.c (262): failed to create worker default
[Fri May 20 14:07:00.043 2011] [12064:14296] [error]
uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with
name 'lbaspgrp14' in uri map post processing.
… (hundreds of these messages) …
[Fri May 20 14:07:00.606 2011] [12064:14296] [error]
uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with
name 'lbaspgrp13' in uri map post processing.
[Fri May 20 14:07:00.638 2011] [12064:14904] [error]
ajp_worker_factory::jk_ajp_common.c (2929): allocating ajp worker record
from shared memory
[Fri May 20 14:07:00.638 2011] [12064:14904] [error]
wc_create_worker::jk_worker.c (151): factory for ajp13 failed for default
[Fri May 20 14:07:00.653 2011] [12064:14904] [error]
build_worker_map::jk_worker.c (262): failed to create worker default
[Fri May 20 14:07:00.653 2011] [12064:14904] [error]
uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with
name 'lbaspgrp14' in uri map post processing.
… (hundreds of these messages) …
[Fri May 20 14:07:01.169 2011] [12064:14904] [error]
uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with
name 'lbaspgrp13' in uri map post processing.
Beginning at 14:06:59 and continuing until we shut down IIS and failed over
to another server 30 minutes later, requests began appearing in the IIS log
file again, but at this time ALL requests were replied to with a 500
internal server error. We have been running this server with this
configuration for over a year without any issues like this until now. No
configuration settings were changed. No workers were added. We do not have
shm_size set anywhere since, per the documentation, that directive is not
needed anymore as of 1.2.27.
I have been unable to find any useful information about this elsewhere, save
this bug:
https://issues.apache.org/bugzilla/show_bug.cgi?id=40877
However, that bug was apparently fixed back in 2007 with version 1.2.20. So,
I’m not sure if it’s related or not.
Any insights? Has anyone else seen this? Is this an IIS problem or a
new/reoccurring JK Connector bug or a configuration problem?
Thanks in advance!
Nick