ajp_get_reply, ajp_connection_tcp_get_message Errors

[Alan <du...@gmail.com>]

I have been trouble-shooting a problem that appears constantly in my
mod_jk.log but is
only visible intermittently to users for a few months now.  Searching the
web and this archive
for similar error messages, I have found some solutions that have led me to
better Apache and
Tomcat setups, but have not solved the problem.  The problem is that a
commercial application
we are serving via Apache/Tomcat will periodically display a "Service
Temporarily Unavailable"
message in a random frame within the application, refreshing the browser
usually allows the
users to recover but this becomes painful when users go through the
multi-step process of entering data
or uploading files only to be interrupted by this and have to start over.
Support for the
commercial application has not been particularly helpful so I am hoping to
find suggestions
here to further try to solve this problem.

Our setup is Apache 2.0.59 (Win32) mod_ssl/2.0.59 OpenSSL/0.9.8d
mod_jk/1.2.19 and
Tomcat 5.5.15 running on Windows Server 2003 R2 Service Pack 1.  I will
include httpd.conf,
ssl.conf, workers.properties, and server.xml below.

Regardless of whether or not users see the "Service Temporarily Unavailable"
message in
any of the frames within the application, my mod_jk.log is full of errors
like this:

[Fri Apr 27 09:38:14 2007] [2764:1792] [error]
ajp_get_reply::jk_ajp_common.c (1536): (myworker) Tomcat is down or refused
connection. No response has been sent to the client (yet)
[Fri Apr 27 09:38:22 2007] [2764:3596] [error]
ajp_connection_tcp_get_message::jk_ajp_common.c (947): (myworker) can't
receive the response message from tomcat, network problems or tomcat is down
(127.0.0.1:8009), err=-54
[Fri Apr 27 09:38:22 2007] [2764:3596] [error]
ajp_get_reply::jk_ajp_common.c (1536): (myworker) Tomcat is down or refused
connection. No response has been sent to the client (yet)
[Fri Apr 27 09:38:25 2007] [2764:2148] [error]
ajp_get_reply::jk_ajp_common.c (1536): (myworker) Tomcat is down or refused
connection. No response has been sent to the client (yet)
[Fri Apr 27 09:38:31 2007] [2764:1792] [error]
ajp_connection_tcp_send_message::jk_ajp_common.c (914): sendfull returned
-53 with errno=53
[Fri Apr 27 09:38:31 2007] [2764:1792] [error]
ajp_send_request::jk_ajp_common.c (1350): (myworker) error sending request
body

Setting my log level to debug does not tell me much more:

[Tue May 15 14:21:38 2007] [1804:3328] [debug] ajp_service::jk_ajp_common.c
(1734): processing myworker with 2 retries
[Tue May 15 14:21:38 2007] [1804:3328] [info]
ajp_send_request::jk_ajp_common.c (1170): (myworker) socket 748 is not
connected any more (errno=0)
[Tue May 15 14:21:38 2007] [1804:3328] [info]
ajp_send_request::jk_ajp_common.c (1194): (myworker) error sending request.
Will try another pooled connection
[Tue May 15 14:21:38 2007] [1804:3328] [info]
ajp_send_request::jk_ajp_common.c (1170): (myworker) socket 784 is not
connected any more (errno=0)
[Tue May 15 14:21:38 2007] [1804:3328] [info]
ajp_send_request::jk_ajp_common.c (1194): (myworker) error sending request.
Will try another pooled connection
[Tue May 15 14:21:38 2007] [1804:3328] [info]
ajp_send_request::jk_ajp_common.c (1218): (myworker) all endpoints are
disconnected or dead
[Tue May 15 14:21:38 2007] [1804:3328] [info]  ajp_service::jk_ajp_common.c
(1867): (myworker) sending request to tomcat failed,  recoverable operation
attempt=1

As these are the only errors I see in any of my log files, I am convinced
this is why users will
periodically see frames that think Tomcat is not responding, why users do
not see these unavailable
errors in frames every time it occurs in the mod_jk.log is a little beyond
me.  We are tied to this
version of Tomcat (5.5.15) and to Java 1.5.0_05 by the commercial
application we are running, for support
purposes, even though their support has been unable to help us.

Any help I can get to at least resolve the errors I see in the mod_jk.log
would be great or if you see any
other errors in my configuration, although I think I have corrected it
pretty well.  See below.

Thanks

Brice

workers.properties
#####################################
workers.tomcat_home=D:\Tomcat5515
workers.java_home=D:\jdk1.5.0_05
ps=/
worker.list=myworker

worker.myworker.type=ajp13
worker.myworker.host=localhost
worker.myworker.port=8009
worker.myworker.cachesize=10
worker.myworker.cache_timeout=600
worker.myworker.socket_keepalive=1
worker.myworker.recycle_timeout=30
worker.myworker.lbfactor=1
#####################################

ssl.conf
#####################################
<IfDefine SSL>
    Listen 443
    AddType application/x-x509-ca-cert .crt
    AddType application/x-pkcs7-crl    .crl
    SSLPassPhraseDialog  builtin
    SSLSessionCache         dbm:logs/ssl_scache
    SSLSessionCacheTimeout  1500
    SSLMutex default
    SSLRandomSeed startup builtin
    SSLRandomSeed connect builtin

</IfDefine>
#####################################

httpd.conf snippet that I feel is relevant, I can post more if needed
#####################################
<IfModule mod_ssl.c>
    Include conf/ssl.conf
</IfModule>
<IfModule mod_jk.c>
    JkWorkersFile conf/workers.properties
    JkLogFile logs/mod_jk.log

    # Set the jk log level [debug/error/info]
    JkLogLevel debug
    JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "

    # JkOptions indicate to send SSL KEY SIZE,
    JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories
    # JkRequestLogFormat set the request format
    JkRequestLogFormat "%w %V %T"
</IfModule>
NameVirtualHost xxx.xxx.xxx.xxx:443
<IfDefine SSL>
    <VirtualHost xxx.xxx.xxx.xxx:443>
        ServerName mytomcatservername.net
        <Directory />
            Options FollowSymLinks -Indexes
            AllowOverride none
        </Directory>
        SSLEngine on
        SSLCertificateFile conf/ssl.crt/server.crt
        SSLCertificateKeyFile conf/ssl.key/server.key

        JkMount /applicationname/*.jsp myworker
        JkMount /applicationname/servlet/* myworker
        JkMount /applicationname/servlet myworker
    </VirtualHost>
</IfDefine>
#####################################

server.xml
#####################################
<Server port="8005" shutdown="SHUTDOWN">
  <Listener className="org.apache.catalina.core.AprLifecycleListener" />
  <Listener className="org.apache.catalina.mbeans.ServerLifecycleListener"
/>
  <Listener className="
org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
  <Listener className="
org.apache.catalina.storeconfig.StoreConfigLifecycleListener"/>


  <GlobalNamingResources>

    <Environment name="simpleValue" type="java.lang.Integer" value="30"/>

    <Resource name="UserDatabase" auth="Container"
              type="org.apache.catalina.UserDatabase"
       description="User database that can be updated and saved"
           factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
          pathname="conf/tomcat-users.xml" />

  </GlobalNamingResources>

  <Service name="Catalina">

    <Connector port="8009"
               enableLookups="false" redirectPort="8443"
connectionTimeout="0" protocol="AJP/1.3" />

    <Engine jvmRoute="ematrix" name="Catalina" defaultHost="localhost">
      <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
             resourceName="UserDatabase"/>
      <Host name="localhost" appBase="webapps"
       unpackWARs="true" autoDeploy="true"
       xmlValidation="false" xmlNamespaceAware="false">

      </Host>

    </Engine>

  </Service>

</Server>
#####################################