Internal Server Errors while/after restarting Tomcat with mod_jk

[Jan Ploski <jp...@gmx.de>]

Hi,

I have two questions regarding mod_jk's way of dealing with Tomcat
going away.

(1)

Can mod_jk be extended safely to retry connecting for a certain
time instead of immediately reporting an Internal Server Error if
a Tomcat process is temporarily unavailable? I have modified the
function jk_open_socket to do just that (keep trying for a minute
in 2 second intervals before giving up). However, I am not sure
if this is the correct place and whether it would have negative
side effects in some configurations (for example, I can imagine
that in a clustered setup, mod_jk should not wait for Tomcat to
recover, but fail over to another instance). Any ideas whether this
feature could be made optional or what a better place to implement
it would be?

The purpose of all that is to make Tomcat restarts (which are inevitable,
as we develop our applications; automatic class reloading doesn't always
do the trick) transparent to the web server.

(2)

I observed that stopping and restarting Tomcat while Apache with mod_jk
is running results in Internal Server Errors starting to appear randomly
in further requests. I guess that this is caused by mod_jk trying to
use a now dead Tomcat instance to service a request. However, I looked
into mod_jk code (jk_ajp13_worker.c) and it seems that it should attempt
reconnecting if it detects a bad socket. Perhaps it ought to also remove
the worker from its pool and fail over to the next available one instead
of spitting out an Internal Server Error. I am not sure because I don't
know how workers announce their presence to mod_jk and have not found
any tech docs for mod_jk besides of install/configuration issues.
Here is a snippet from the log illustrating what happens:

[jk_uri_worker_map.c (344)]: Into jk_uri_worker_map_t::map_uri_to_worker
[jk_uri_worker_map.c (406)]: jk_uri_worker_map_t::map_uri_to_worker, Found a match ajp13
[jk_worker.c (123)]: Into wc_get_worker_for_name ajp13
[jk_worker.c (127)]: wc_get_worker_for_name, done  found a worker
[jk_ajp13_worker.c (651)]: Into jk_worker_t::get_endpoint
[jk_ajp13_worker.c (536)]: Into jk_endpoint_t::service
[jk_ajp13.c (346)]: Into ajp13_marshal_into_msgb
[jk_ajp13.c (480)]: ajp13_marshal_into_msgb - Done
[jk_ajp13_worker.c (606)]: Error sending request body
[jk_ajp13_worker.c (489)]: Into jk_endpoint_t::done

Can someone confirm that restarting a Tomcat worker is unsupported by
mod_jk and will lead to problems? If so, could someone knowledgeable
in this code advise how to work around this problem or whether it would
be a major effort to implement a transparent failover? It seems silly
to restart Apache every time (one of) Tomcats needs to be reinitialized,
just for mod_jk's sake.

Thanks in advance for any hints.

-JPL

Re: Internal Server Errors while/after restarting Tomcat with mod_jk

[Dan Milstein <da...@shore.net>]

Jan,

Henri Gomez is working on a patch to solve this very problem as we speak --
he is setting it up so that the mod_jk plugin will detect that TC has died,
and will attempt to open new connections.

Hopefully, that will be testable within a week or two...

-Dan


Jan Ploski wrote:
> 
> Hi,
> 
> I have two questions regarding mod_jk's way of dealing with Tomcat
> going away.
> 
> (1)
> 
> Can mod_jk be extended safely to retry connecting for a certain
> time instead of immediately reporting an Internal Server Error if
> a Tomcat process is temporarily unavailable? I have modified the
> function jk_open_socket to do just that (keep trying for a minute
> in 2 second intervals before giving up). However, I am not sure
> if this is the correct place and whether it would have negative
> side effects in some configurations (for example, I can imagine
> that in a clustered setup, mod_jk should not wait for Tomcat to
> recover, but fail over to another instance). Any ideas whether this
> feature could be made optional or what a better place to implement
> it would be?
> 
> The purpose of all that is to make Tomcat restarts (which are inevitable,
> as we develop our applications; automatic class reloading doesn't always
> do the trick) transparent to the web server.
> 
> (2)
> 
> I observed that stopping and restarting Tomcat while Apache with mod_jk
> is running results in Internal Server Errors starting to appear randomly
> in further requests. I guess that this is caused by mod_jk trying to
> use a now dead Tomcat instance to service a request. However, I looked
> into mod_jk code (jk_ajp13_worker.c) and it seems that it should attempt
> reconnecting if it detects a bad socket. Perhaps it ought to also remove
> the worker from its pool and fail over to the next available one instead
> of spitting out an Internal Server Error. I am not sure because I don't
> know how workers announce their presence to mod_jk and have not found
> any tech docs for mod_jk besides of install/configuration issues.
> Here is a snippet from the log illustrating what happens:
> 
> [jk_uri_worker_map.c (344)]: Into jk_uri_worker_map_t::map_uri_to_worker
> [jk_uri_worker_map.c (406)]: jk_uri_worker_map_t::map_uri_to_worker, Found a match ajp13
> [jk_worker.c (123)]: Into wc_get_worker_for_name ajp13
> [jk_worker.c (127)]: wc_get_worker_for_name, done  found a worker
> [jk_ajp13_worker.c (651)]: Into jk_worker_t::get_endpoint
> [jk_ajp13_worker.c (536)]: Into jk_endpoint_t::service
> [jk_ajp13.c (346)]: Into ajp13_marshal_into_msgb
> [jk_ajp13.c (480)]: ajp13_marshal_into_msgb - Done
> [jk_ajp13_worker.c (606)]: Error sending request body
> [jk_ajp13_worker.c (489)]: Into jk_endpoint_t::done
> 
> Can someone confirm that restarting a Tomcat worker is unsupported by
> mod_jk and will lead to problems? If so, could someone knowledgeable
> in this code advise how to work around this problem or whether it would
> be a major effort to implement a transparent failover? It seems silly
> to restart Apache every time (one of) Tomcats needs to be reinitialized,
> just for mod_jk's sake.
> 
> Thanks in advance for any hints.
> 
> -JPL

-- 

Dan Milstein // danmil@shore.net