internal server errors with Apache-Tomcat-JK2 connections

[Tom Hickerson <th...@akazaresearch.com>]

Hi everyone,
 
Stop me if you've heard this one before.  No seriously, stop me because I've had this problem for over a week now, and need to get it solved.  Any suggestions would be appreciated.
 
I am running a servlet-based application on Tomcat 4.1.27 together with Apache 2.0.51 on RH FC2, using mod_jk2 as a connector between the two servers.  Everything runs fine except on certain pages when Tomcat crashes horribly with an Internal Server error and the message is displayed:
 
The servlet container is temporary unavailable or being upgraded 

Internal Server Error


The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, root@localhost and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.

Additionally, a 500 Internal Server Error error was encountered while trying to use an ErrorDocument to handle the request.

Apache's error_log contains the following messages when this is done:

[Sat Jun 10 20:03:41 2006] [error] channelSocket.open() connect failed localhost:8019 111 Connection refused 
[Sat Jun 10 20:03:41 2006] [error] ajp13.connect() failed ajp13:localhost:8019
[Sat Jun 10 20:03:41 2006] [error] ajp13.service() failed to connect endpoint errno=111 Connection refused
[Sat Jun 10 20:03:41 2006] [error] ajp13.service() Error  forwarding ajp13:localhost:8019 1 1
[Sat Jun 10 20:03:41 2006] [notice] ajp13.done() close endpoint ajp13:localhost:8019 error_state 1
[Sat Jun 10 20:03:41 2006] [error] lb.service() worker failed 120000 for ajp13:localhost:8019
[Sat Jun 10 20:03:41 2006] [error] channelUn.connect() connect failed 13 Permission denied
[Sat Jun 10 20:03:41 2006] [error] ajp13.connect() failed ajp13:/usr/share/tomcat/work/jk2.socket
[Sat Jun 10 20:03:41 2006] [error] ajp13.service() failed to connect endpoint errno=13 Permission denied
[Sat Jun 10 20:03:41 2006] [error] ajp13.service() Error  forwarding ajp13:/usr/share/tomcat/work/jk2.socket 1 1
[Sat Jun 10 20:03:41 2006] [notice] channelUn.close(): close unix socket -1 
[Sat Jun 10 20:03:41 2006] [notice] ajp13.done() close endpoint ajp13:/usr/share/tomcat/work/jk2.socket error_state 1
[Sat Jun 10 20:03:41 2006] [error] lb.service() worker failed 120000 for ajp13:/usr/share/tomcat/work/jk2.socket
[Sat Jun 10 20:03:45 2006] [error] channelSocket.open() connect failed localhost:8019 111 Connection refused 
[Sat Jun 10 20:03:45 2006] [error] ajp13.connect() failed ajp13:localhost:8019
[Sat Jun 10 20:03:45 2006] [error] ajp13.service() failed to connect endpoint errno=111 Connection refused
[Sat Jun 10 20:03:45 2006] [error] ajp13.service() Error  forwarding ajp13:localhost:8019 1 1
[Sat Jun 10 20:03:45 2006] [notice] ajp13.done() close endpoint ajp13:localhost:8019 error_state 1
[Sat Jun 10 20:03:45 2006] [error] lb.service() worker failed 120000 for ajp13:localhost:8019
[Sat Jun 10 20:03:45 2006] [error] channelUn.connect() connect failed 13 Permission denied
[Sat Jun 10 20:03:45 2006] [error] ajp13.connect() failed ajp13:/usr/share/tomcat/work/jk2.socket
[Sat Jun 10 20:03:45 2006] [error] ajp13.service() failed to connect endpoint errno=13 Permission denied
[Sat Jun 10 20:03:45 2006] [error] ajp13.service() Error  forwarding ajp13:/usr/share/tomcat/work/jk2.socket 1 1
[Sat Jun 10 20:03:45 2006] [notice] channelUn.close(): close unix socket -1 
[Sat Jun 10 20:03:45 2006] [notice] ajp13.done() close endpoint ajp13:/usr/share/tomcat/work/jk2.socket error_state 1
(...a lot more ajp13 errors follow this...)

 

Just for the record, the jk2.properties file is the following:

## THIS FILE MAY BE OVERRIDEN AT RUNTIME. MAKE SURE TOMCAT IS STOPED
## WHEN YOU EDIT THE FILE.

## COMMENTS WILL BE _LOST_

## DOCUMENTATION OF THE FORMAT IN JkMain javadoc.

# Set the desired handler list
# handler.list=apr,request,channelJni
#
# Override the default port for the socketChannel
# channelSocket.port=8019
# Default: 
# channelUnix.file=${jkHome}/work/jk2.socket
# Just to check if the the config  is working
# shm.file=${jkHome}/work/jk2.shm

# In order to enable jni use any channelJni directive
# channelJni.disabled = 0
# And one of the following directives:

# apr.jniModeSo=/opt/apache2/modules/mod_jk2.so

# If set to inprocess the mod_jk2 will Register natives itself
# This will enable the starting of the Tomcat from mod_jk2
# apr.jniModeSo=inprocess


And our workers2.properties is the following:

[logger]
level=DEBUG

[config:]
file=${serverRoot}/conf/workers2.properties
debug=0
debugEnv=0

[uriMap:]
info=Maps the requests. Options: debug
debug=0

[shm:]
info=Scoreboard. Required for reconfiguration and status with multiprocess servers
#file=${serverRoot}/logs/jk2.shm
#replaced, was the following
#file=/var/log/tomcat/shm.file
file=/var/log/httpd/jk2.shm
size=1000000
debug=0
disabled=0

[workerEnv:]
info=Global server options
timing=1
debug=0
# Default Native Logger (apache2 or win32 ) 
# can be overriden to a file logger, useful 
# when tracing win32 related issues
#logger=logger.file:0

[lb:lb]
info=Default load balancer.
debug=0

[lb:lb_1]
info=A second load balancer.
debug=0

[channel.socket:localhost:8009]
info=Ajp13 forwarding over socket
debug=0
tomcatId=localhost:8009

[channel.socket:localhost:8019]
info=A second tomcat instance. 
debug=0
tomcatId=localhost:8019
lb_factor=1
group=lb
group=lb_1
disabled=0

[channel.un:/var/cache/tomcat/jk2.socket]
# was /opt/33/work/
info=A second channel connecting to localhost:8019 via unix socket
tomcatId=localhost:8019
lb_factor=1
debug=0

# define the worker
[ajp13:/var/cache/tomcat/jk2.socket]
channel=channel.un:/var/cache/tomcat/jk2.socket

[channel.jni:jni]
info=The jni channel, used if tomcat is started inprocess

[status:]
info=Status worker, displays runtime informations

[vm:]
info=Parameters used to load a JVM in the server process
#JVM=C:\jdk\jre\bin\hotspot\jvm.dll
classpath=${TOMCAT_HOME}/bin/tomcat-jni.jar
classpath=${TOMCAT_HOME}/server/lib/commons-logging.jar
OPT=-Dtomcat.home=${TOMCAT_HOME}
OPT=-Dcatalina.home=${TOMCAT_HOME}
OPT=-Xmx128M
#OPT=-Djava.compiler=NONE
disabled=1

[worker.jni:onStartup]
info=Command to be executed by the VM on startup. This one will start tomcat.
class=org/apache/jk/apr/TomcatStarter
ARG=start
# For Tomcat 5 use the 'stard' for startup argument
# ARG=stard
disabled=1
stdout=${serverRoot}/logs/stdout.log
stderr=${serverRoot}/logs/stderr.log

[worker.jni:onShutdown]
info=Command to be executed by the VM on shutdown. This one will stop tomcat.
class=org/apache/jk/apr/TomcatStarter
ARG=stop
disabled=1

[uri:/jkstatus/*]
info=Display status information and checks the config file for changes.
group=status:

[uri:127.0.0.1:8003]
info=Example virtual host. Make sure myVirtualHost is in /etc/hosts to test it
alias=myVirtualHost:8003

[uri:127.0.0.1:8003/ex]
info=Example webapp in the virtual host. It'll go to lb_1 ( i.e. localhost:8019 )
context=/ex
group=lb_1

[uri:/examples]
info=Example webapp in the default context.
context=/examples
debug=0

[uri:/examples1/*]
info=A second webapp, this time going to the second tomcat only.
group=lb_1
debug=0

[uri:/examples/servlet/*]
info=Prefix mapping

[uri:/examples/*.jsp]
info=Extension mapping

[uri:/examples/*]
info=Map the whole webapp

[uri:/examples/servlet/HelloW]
info=Example with debug enabled.
debug=10

[..more uri's here for our applications...]

Again, any advice would be appreciated.  Let me know if you need any more information on the errors or the server setup.

Thanks in advance, 

Tom Hickerson, Akaza Research